Psyko Cybernetik

Drupal Inline Popup Reference Field

Wed, 07 Oct 2009 22:36:00 +0200

It seems that I am spending some time learning Drupal so the content of this post may defer from the editorial line a bit but as usual, I hope it may help someone.

The current drupal installation I am working with has CCK and Popups: Add and Reference (let’s call it PAR for future (shorter) references) modules. Unfortunately, by default the links added by PAR are in a new div tag below the input fields which increase the length of the form quite a bit as you can see below.

What I am proposing here is a way to put the PAR links inline with the input field. I am quite a beginner with Drupal so the method may not be really clean or fail proof and your comments about any easier way of doing it are welcomed. The result looks like the screenshot below.

In order to get to the result, I had to modify the following files:

the module file: popups_reference.module
the template file of my theme: template.php

In the popups_reference.module file (that is located under sites/all/modules/popups_reference), I have modified the function popups_reference_alter_item(&$form, $key, $item, $fields) in order to remove the link addition from the $form[$key]['#suffix']. Instead, I am putting the links in an array that is used later by the post_render function.

The post_render function is actually looking for some string that I have added in the template.php and is replacing it if needed, i.e. if there is a PAR link to output. Let’s take at what was done in the template.php file (located under sites/all/themes/MYTHEME (where MYTHEME is to be replaced by the appropriate name, i.e. your theme name)).

In the template.php, I override the form_element function that is used to display the field titles, description and input. You can see the complete function here under, but the only addition is the following line after the value is added to the $output variable.

  $output .= "<span class=\"popups-reference-link\"/>\n";

/**
 * Function used to overwrite the default display of form elements.
 * The description tag is placed before the input tag
 * A span tag for the popups_reference module has been added before the final div. 
 * It allows to have the links created by the popups_reference module inline with the input field.
 */
function NewsFlash_form_element($element, $value) {
  $output  = '<div class="form-item"';
  if (!empty($element['#id'])) {
    $output .= ' id="'. $element['#id'] .'-wrapper"';
  }
  $output .= ">\n";
  $required = !empty($element['#required']) ? '<span class="form-required" title="'. t('This field is required.') .'">*</span>' : '';

  if (!empty($element['#title'])) {
    $title = $element['#title'];
    if (!empty($element['#id'])) {
      $output .= ' <label for="'. $element['#id'] .'">'. t('!title: !required', array('!title' => filter_xss_admin($title), '!required' => $required)) ."</label>\n";
    }
    else {
      $output .= ' <label>'. t('!title: !required', array('!title' => filter_xss_admin($title), '!required' => $required)) ."</label>\n";
    }
  }

  $output .= " $value\n";

  // Add a span tag that will be replaced if necessary when the popups_reference 
  // module adds links to create new node.
  $output .= "<span class=\"popups-reference-link\"/>\n";

  if (!empty($element['#description'])) {
    $output .= ' <div class="description">'. $element['#description'] ."</div>\n";
  }

  $output .= "</div>\n";

  return $output;
}

And that’s it to display the “Add New: Add XXX” link inline with the input or select field.

I have attached the modified popups_reference.module and my template.php to this post. If you have any comment regarding a better way to do that, I’d be glad to know, so feel free to drop a comment.

<?php
// $Id: popups_reference.module,v 1.1.2.12 2009/03/07 06:54:25 starbow Exp $

/**
 * @file
 * Modify the Node Reference widget to use a popup to add a new node.
 */ 

$links_replacement;

/**
 * Implementation of hook_form_alter().
 * 
 * Modifies the nodereference setting form and the basic node form. 
 */
function popups_reference_form_alter(&$form, $form_state, $form_id) {
  if ($form_id == 'content_field_edit_form' && $form['#field']['type'] == 'nodereference') {
    // Add a checkbox to the nodereference settings page.
    $field_name = $form['#field']['field_name'];
    $form['field']['show_add_link'] = array(
    '#type' => 'checkbox',
      '#default_value' => variable_get('popups_reference_show_add_link_'. $field_name, TRUE),
      '#title' => t('Show the "Add New: Node Type" Popup links'),
      '#description' => t("Activate Popups:Add and Reference behavior for this reference.")
    );
    $form['#submit'][] = '_popups_reference_manage_fields_submit';
  }
  elseif (isset($form['type'])) {
    // Add the "Add New: Node Type" links.
    $node = $form['#node'];
    if ($form['type']['#value'] .'_node_form' == $form_id) {  
      $fields = content_fields();
      foreach ($form as $key => $item) {
        if (is_array($item)) {
          $type = $item['#type'];
          if ($type == 'fieldset') { // Loop through all the subitems.
            foreach ($form[$key] as $subkey => $subitem) {
              popups_reference_alter_item($form[$key], $subkey, $subitem, $fields);
            }
          }
          else {
            popups_reference_alter_item($form, $key, $item, $fields);
          }
        }

      }
    }
  }
}

/**
 * Implementation of hook_nodeapi().
 * Add cookies with node info when a new node is created.
 * These cookies will be found by the popups_reference behavior and used
 *   to select the newly created node in the reference widget.
 */ 
function popups_reference_nodeapi($node, $op) {
  if ($op == 'insert') {
      $five = time()+300; // 5 minutes in the future.
      setcookie("PopupRefNid", $node->nid, $five, '/'); 
//      setcookie("PopupRefTitle", $node->title, $five, '/');
      setrawcookie("PopupRefTitle", rawurlencode($node->title), $five, '/');
  }
}

/**
 * Submit added to the the nodereference settings form.
 * Set a variable for each nodereference field.
 */
function _popups_reference_manage_fields_submit($form, &$form_state) {
  $field_name = $form['#field']['field_name'];
  variable_set('popups_reference_show_add_link_'. $field_name, $form_state['values']['show_add_link']);
}

/**
 * Run on every element in the basic node form.
 * Wrap the enabled nodereference fields, and add the popup links.
 *
 * @param $form - the form (or fieldgroup).
 * @param $key - form element name.
 * @param $item - the form element array.
 * @param $fields - all fields info.
 */
function popups_reference_alter_item(&$form, $key, $item, $fields) {  
  $field_name = strstr($key, 'field_'); // Check if $key starts with 'field_';
  if (isset($fields[$field_name]) && 
      $fields[$field_name]['type'] == 'nodereference' &&
      variable_get('popups_reference_show_add_link_'. $field_name, TRUE)) {
    $type = $form['type']['#value'];
    $field = content_fields($field_name, $type);
    $wrapper_id = 'popups-reference-' . _popups_reference_counter();
    $links = _popups_reference_links($field, $type, $wrapper_id, $field['widget']['type']);
    if ($links) {
      // Put the nodereference widget and links in an wpopups link cssrapper.
      // Makes it easy to find for Ahah targeting, and popups_reference behavior selecting.
      global $links_replacement;
      // Register the links into the global array. Key is the field name so that we can identify the 
      // correct element in the post_render method
      $links_replacement[$field_name] = implode(', ', $links);
      // Set prefix and suffix
      $form[$key]['#prefix'] = '<div id="'. $wrapper_id .'">';
      $form[$key]['#suffix'] = '</div>';
      // Set the post render method that will be called when the variable are rendered
      $form[$key]['#post_render'] = array('cck_field_post_render_popups_reference_link');
    }
  }
}

// Post_render method
function cck_field_post_render_popups_reference_link($content, $element){
	global $links_replacement;	
	//dsm($element);
	foreach(	$links_replacement as $field_name => $links) {
		// If the content contains the field_name, then we can display the links
		if(strstr($content, $field_name)){
			$tag_to_replace = "<span class=\"popups-reference-link\"/>";
			$replacing_tag_prefix = '<span class="popups-reference-link"> Créer: ';
			$replacing_tag_suffix = '</span>';
			// If the type of nodereference is an input or a select, then we can replace all (i.e. the only one) span tag existing
			if ($element['#type'] != 'nodereference_buttons'){
				$content = str_replace($tag_to_replace, $replacing_tag_prefix . $links . $replacing_tag_suffix, $content);
			}
			// If the type of nodereference is a list of checkboxes, then we replace only the last span tag
			// i.e the tag that appears after all the checkboxes
			else {
				$last_occurrence = strrpos($content, $tag_to_replace);
				$content = substr_replace($content, $replacing_tag_prefix . $links . $replacing_tag_suffix, $last_occurrence, $last_occurrence + strlen($tag_to_replace) - strlen($content));
			}		
		}
	}
	return $content;
}

/**
 * Generates 'Add new...' link
 * for each allowed content type
 *
 * @param $field
 * @param $src_type - the type of base node.
 * @param $wrapper_id - id for the wrapper around the node reference.
 * @param $type - the type of widget.
 * @return Array of html links.
 */
function _popups_reference_links($field, $src_type, $wrapper_id, $widget_type) {
  if ($widget_type == 'nodereference_select' || $widget_type == 'nodereference_buttons') { 
    // Target the wrapper for replacing.
    popups_add_popups(array('a.'.$wrapper_id=>array('targetSelectors'=>array('#'.$wrapper_id))));
  }
  else if ($widget_type == 'nodereference_autocomplete') { 
    // Don't replace the autocomplete when done.
    popups_add_popups(array('a.'.$wrapper_id=>array('noUpdate'=>TRUE)));
  }
  else { // Unsupported type.
    return;
  }
  $options = array(
    'attributes' => array(
      'class' => $wrapper_id . ' popups-reference', 
      'rel' => $wrapper_id,
  ),
    'query' => array('destination' => 'node/add/' . str_replace('_', '-', $src_type)),  
  );
  $links = array();
  $all_types = node_get_types();
  foreach ($field['referenceable_types'] as $add_type => $value) {
    if (!empty($value) && (user_access("create $add_type content") || user_access('administer nodes'))) {
    //if (!empty($value) && user_access("create $add_type content")) {
      drupal_add_js(drupal_get_path('module', 'popups_reference') .'/popups_reference.js');
      $path = 'node/add/' . str_replace('_', '-', $add_type);
      $name = $all_types[$add_type]->name;
      $links[] = l(" $name", $path, $options);
    }
  }
  return $links;
}

/**
 * A counter for generating unique element id's.
 *
 * @return int: next integer.
 */
function _popups_reference_counter() {
  static $count = 0;
  return $count++;
}

Drupal Private Download Folder

Wed, 26 Aug 2009 23:16:00 +0200

Still working with Drupal and still learning a lot… The challenge I was facing was to let users upload files to a specific folder but to restrict access to that file so that it cannot be downloaded by anybody simply by giving the file path (private download). Therefore, users are allowed to post content but not access it (otherwise nothing prevent them to access files from other users in the folders).

Fortunately - as often with Drupal - there are already solutions on the web:

To summarize, I mixed the two solutions given here above, because I had a problem when using only solution 1 (my drupal site is not at the root of the website but in a subfolder). Therefore I used method 2 and edited the .htaccess file at the root of the website in order to add the following line somewhere in the block delimited by <IfModule mod_rewrite.c>...</IfModule>:

RewriteRule ^sites\/default\/files\/(protected_download_dir\/.*)$ index.php?q=system/files/$1

For what I have noticed, it is also possible to edit put the line as

RewriteRule ^sites\/default\/files\/(protected_download_dir\/.*)$ /system/files/$1

but maybe one is better than the other.

After that, I just implemented the private download module given in one of the comment of the first link. As a result, the following is now the current flow:

Some users are allowed to create a custom content containing a CCK filefield item so that they can upload a file in the private folder
Once the file is uploaded, normal users do not have the permission to access the file
Administrators receive a email that a new content containing a file has been uploaded
Administrator with the privatedownload permission (part of the module) are allowed to access the file in the private folder and therefore allowed to download it

Note that as mentioned in the links above, this solution offers one private folder that would be like a pool in which user can throw their files but not access them. Only administrators are allowed to access the files. It is therefore easier if all files are situated in this unique folder so that administrators can retrieve / remove all files at once (by ftp/sftp/ssh) if necessary.

Share Firefox Profile Between Computers

Tue, 07 Jul 2009 22:46:00 +0200

It is no secret, it is easy to share a firefox profile. I am running a dual boot Linux / Windows and was therefore interested to share my Firefox profile between the two. But in the process, I actually became interested to be able to access that profile from my work computer. After all, there are already extensions to store bookmarks remotely so why not do the same with the complete profile.

This is the reason why I decided to create a new profile on my Dropbox. That way, the profile is stored remotely and synchronized in real-time between my work computer and my personal computer (both Linux and Windows).

Creating a new profile can be done in two ways. The first is by using the graphical interface and the second is by modifying the profiles.ini file directly. I will present both.

Add a profile with UI

To access the profile manager,

Under Linux: Open a console and type “firefox -profilemanager” (without the “”)
Under Windows: Open a command line (Windows+R), navigate to the firefox folder (cd C:\Program Files\Mozilla Firefox), and type “firefox -profilemanager”

This will open the profile manager window that should look like the following. In here, simply click the “Create Profile…” button.

The new windows allows to specify the folder where you want to store the profile.

Once the new profile is created, next time firefox start, the profile manager will pop-up and ask the profile that you want to use.

Note that if you want to completely replace your current profil with the “Dropbox” profile, you can simply copy and paste the content of the old default folder to the new “Dropbox” folder and then delete the default profile from the profile manager.

Modify directly the profiles.ini file

As an alternative to the UI method presented above, you can modify directly Firefox initialization file. To do so, you need to locate the file called profiles.ini. The path to the file is as follow:

Under Linux: /home/USERNAME/.mozilla/firefox/
Under windows: C:\Documents and Settings\USERNAME\Application Data\Mozilla\Firefox\

Open the file with your favorite text editor, and simply add a profile. After the addition of my dropbox profile, the file looks like the following:

The StartWithLastProfile indicates whether or not Firefox should ask you at start-up which profile to use or if it should load the previous profile that was in use. The first profile is my default profile and the second profile is the Dropbox profile. To switch to the whole dropbox solution, you just need to remove the default profile.

It is of course possible to move the content of your previous default profile to the new dropbox profile so that all previous information are copied.

A word about security

As anything that is stored on the cloud, be careful about what you put out there. By default, if you have allowed your password to be stored by Firefox without any master password, they are stored in clear. I actually haven’t checked where Firefox stores the passwords as I am not using it to store my passwords but I wouldn’t be surprised if it is stocked somewhere in the profile folder. So even though your Dropbox folder is private and probably encrypted on the server, better be safe than sorry. So use other password manager (like keypass for instance). You’ve been warned and can’t blame me if anything goes wrong ;-)

Last comment

Ok, this post was no rocket science :-). However, it gives a good overview of what you can do with the so-called “cloud” in order not only to keep your documents synchronized but your preferences. I am sure that you can use the method with other softwares but as mention previously, always keep the security in mind! Find out if the application store any password or personal information before adventuring yourself with such a solution.

Based on the scenario presented above, I am sure that one could easily put a more secure solution in place. One could for instance store the profile in an encrypted container (with truecrypt) that would be mounted at start-up. Using Dropbox is not the only solution of course. One could use his own folder anywhere on the web mounted as a network drive over ssh. I am sure that you can come up with many ideas about how to generalize the suggested idea…

Install Linux Mint on Lenovo X200

Mon, 06 Jul 2009 18:30:00 +0200

In my previous post, I detailed how I partitioned the drive of the X200 in order to install Linux. I will now give a few words about the installation process of Linux Mint 7. The focus of this post is set on how to keep your boot loader as it is and allow a dual boot windows/Linux from the Windows boot loader.

There are only a few things that I changed from the default installation. The first one concerns the partitioning. Since the drive is already partitioned, it is necessary to tell the installer which partitions to use for what. From the picture below, you can easily see which partitions are used and where they are mounted.

The important thing is to mount /boot on the primary partition created previously (/dev/sda4 in the example above). I decided to create a partition for /home as well so that a future upgrade of the system doesn’t overwrite personal data. This is a good common practice unless upgrade means global cleaning to you.

During the last step of the installation process, it is important to click on the advanced button if you don’t want Grub to be installed on the MBR. As stated in introduction, I want to keep the MBR so I specified that Grub should be installed on the primary partition that was created (/dev/sda4).

When installation is finished, restart the computer. There is at the moment no possibility to boot Linux as the MBR as not be replaced and no entry has been added to Windows boot file. It is therefore necessary first to reboot the live CD so that we can copy Linux boot sector. To do so, open a terminal and enter the following command:

dd if=/dev/sda4 of=/media/disk/linux.bin bs=512 count=1

In the command line above, don’t forget to indicate the correct paths to the if and of parameters so that it reflects your installation. Note that in the example above I am copying the boot sector to a USB stick and it is necessary afterwards to copy from the USB stick to the C drive of windows.

When the boot sector is copied, we can copy it in a windows accessible folder and edit C:\boot.ini.If there is any problem to edit it, don’t forget to remove the Read-Only flag by right click > Properties. My boot.ini looks like the following:

[boot loader]
timeout=15
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect
c:\linux.bin="Linux-Mint"

You can specify the timeout and default system to start by modifying the line below [boot loader]. You will notice that when you chose the Linux-Mint entry, Grub will be called offering you some more options.

If you want more detailed instructions concerning the dual boot procedure, I can recommend the following articles:

Partitioning Lenovo X200 to Install Linux

Mon, 06 Jul 2009 00:08:00 +0200

I wanted to install Linux on my computer (Lenovo X200) but I was a bit afraid to mess with the partition table due to the fact that there is a hidden Restore & Recovery partition available at boot up when one press on the ThinkVantage button.

I did want to keep Windows available as there are a few programs that can come handy and I didn’t want to mess with the default behavior of the computer (understand that the ThinkVantage button should still work, that windows should still be accessible and that I can basically still use the laptop the way I used to do :-)).

I will explain in the following paragraphs how to reformat the drive using open source tools and how I created the new partitions for the X200.

Download UNetbootin and PartedMagic and install it on a USB stick.

Make sure that you computer can boot on a USB stick (most recent computer can if the option is set up properly in the BIOS) and make sure that your USB stick is bootable. You can as well install it on the hardrive but if it boots from the harddrive, it may not allow to reformat the drive.
Partition the disk

I have divided the process in two parts:
- Resize the Windows partition (followed by a reboot to verify that windows AND the recovery partition are accessible)
- Create the new partitions
Before resizing the windows partition, do not forget to defragment the disk so that there is no risk of data loss and to create a backup of your data. To resize, just boot on the USB stick and execute GParted. The interface is quite intuitive and many good tutorials exist online. Resizing is the easy part and can take a while so just be patient. Once resized, restart the computer and you will notice that Windows does a check of the drive at startup. The only problem I noticed was related to the icon of the C drive but I will come to that later.

Now that we have some unused space, we can create the partitions that we want (reboot on the USB stick one more time to access GParted). The number of primary partitions is limited to 4 so we will create one extended partition (that will contain many logical partitions) and one primary partition. We need the primary partition in order for Linux to boot without installing the bootloader on the Master Boot Record (MBR).

The partitions I have created can be easily seen on the picture above but here is a quick explanation of the steps:
- Create an extended partition (just leave around 200 MB at the end)
- Create an primary partition in the last 200 MB that you left in the first step
- Within the extended partition, create logical partitions. I have created the following:
  - Two NTFS partitions
  - One partition that will be for Linux root
  - One partition for Linux home
  - One partition for swap
Next time Windows restart, there is two more drives available but as mentioned, the only problem was related to the icon of the C drive. In order to fix it, launch TweakUI and go the Repair section to rebuild icons. Restart the computer and everything should be back to normal.

Create SOCKS Proxy to Bypass Censorship

Wed, 17 Jun 2009 11:29:00 +0200

Using the method described in my previous article, you can easily see the benefit to bypass censorship. By using the set up previously described, you can encrypt all your traffic through the proxy and therefore appear as if you are only having one long SSH connection.

But in addition, it can be a good idea to configure firefox to use the proxy to resolve the DNS requests (in case the DNS server you are using are “filtered”). To do so, type “about:config” in the adress bar and change the setting of network.proxy.socks_remote_dns from false to true as shown below.

Access Pandora Outside the USA

Tue, 16 Jun 2009 22:59:00 +0200

If you enjoy Pandora or any content that is restricted to the USA, it is quite annoying when you cannot access it because you are out of the country (Ha! Good old Europe).

Fortunately, proxies can help you fix the problem quite easily. I used to employ GPass. It is an easy solution to use under Windows but last time I tried to start it, it couldn’t find any tunnel. There is however an alternative solution to put in place if you have a web host in the USA with SSH connection: create your own proxy tunnel.

The process is simple:

Create a SSH tunnel
Configure your web browser to use the tunnel.

Create a SSH tunnel

To create a tunnel, open a console (if you are using Windows, you can use Mobaxvt that I described here) and enter the following:

ssh -ND localhost:5555 user@host.com

The options are explained below but you can have more details here.

-N      Do not execute a remote command.
-D port
	Specifies a local "dynamic" application-level port forwarding.
user@host.com
	Your ssh credentials and webhost address

In other words, we open a remote session and traffic will be redirected to port 5555 of our machine.

Configure your web browser

I am using firefox with the FoxyProxy extension. This extension allows to use different proxy settings depending on the websites that you are visiting. In other terms, not all your traffic need to go through your webhost… only Pandora (and whatever else you feel like).

Once the new proxy is created, configure the proxy as a SOCKS proxy v5, with the configuration given above (address: localhost, port:5555) as illustrated in the screenshot below.

Then, you can configure foxyproxy to use patterns as shown below and you should now have access to Pandora (or whatever you configure) from wherever you are.

SSH Console Under Windows

Thu, 11 Dec 2008 00:06:00 +0100

While looking for an easy (and immediate) way to get an SSH console to windows, I found MobaXVT. It is describe as a “Free portable X server with Unix/Cygwin utilities”. As the description suggest, it is actually a Cygwin encapsulation into a nice multi-tab interface that has a built-in (among other things that I didn’t test) SSH client. Anyway, if you are just looking to have an SSH client under Windows, this is a great solution if you are allergic to the Cygwin installation. This is available for free download at http://mobaxvt.mobatek.net/en/.

SSH with PAM and private / public key authentication on Lacie Edmini

Thu, 17 Apr 2008 00:04:00 +0200

Following the article where I explain how to install a SSH server on the Lacie Edmini, I will explain how to allow authentication through the use of private / public key so that you can use the method explain in this article to backup your files on your local server.

During the installation of the SSH server, we didn’t touch anything in the SSH configuration files. The result was that you could login with the root user you created during the process. The first thing I want to do is to allow a normal user to use ssh. Doing so is easy. Just open the /etc/passwd file and modify the line with the user you want to allow so that it finishes by /bin/bash or /bin/sh depending on the shell you prefer. Finally, a user allowed to connect with ssh will have a line look like:

normalUser:x:503:100:Linux User,,,:/home/normalUserDirectory:/bin/bash

The other difference is the home directory that I modified to /home/userNameDirectory instead of just /home. This step is necessary to create a directory on which the user has full rights and therefore can add and modify everything he wants. With your root user ssh access, do

mkdir /home/normalUserDirectory #create user directory
chown 503 /home/normalUserDirectory #change owner so that it is the same as in /etc/passwd
chgrp 100 /home/normalUserDirectory #change group so that it is the same as in /etc/passwd 
chgrp 100 /home # change group so that it is the same as your user 
chmod 750 /home

Changing the permission of the home directory is required by ssh so that the user will be allowed to connect using his private key. You then need to create a .ssh directory under /home/normalUserDirectory

mkdir /home/normalUserDirectory/.ssh

and change the permissions as we just did before.

chown 503 /home/normalUserDirectory/.ssh
chgrp 100 /home/normalUserDirectory/.ssh
chmod 700 /home/normalUserDirectory/.ssh

In your computer (Linux, Windows with Cygwin, Window with Putty), generate the keys that we will need for authentication. In a previous post, I used a dsa key:

ssh-keygen -b 1024 -f identity -P '' -t dsa

but you could use as well a rsa key:

ssh-keygen -b 2048 -f identity -P '' -t rsa

There are different ways to do it but what’s important is that you verify that the identity.pub that is generated and contains the public key has everything on one line. Verify that the user name at the end of the line is the same that the one you want to allow on the server (i.e. normalUser). Once you have ensure that the file is correct, you can transfer it to the server in the .ssh directory that we have created earlier:

scp identity.pub new_root_user@Lacie_IP_address:/home/normalUserDirectory/.ssh/authorized_keys

And then don’t forget to change owner and group of the authorized_keys file and permissions

chown 503 /home/normalUserDirectory/.ssh/authorized_keys
chgrp 100 /home/normalUserDirectory/.ssh/authorized_keys
chmod 644 /home/normalUserDirectory/.ssh/authorized_keys #you can specify 400 as well

For an alternative transfer method, look at the previous post reference above but one more time don’t forget to set the correct owner and group.

Everything is in place to use the identification with private / public key on the server. The last thing to do is to verify your /etc/ssh/sshd_config file so that it looks like the following:

#    $OpenBSD: sshd_config,v 1.74 2006/07/19 13:07:10 dtucker Exp $
# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options change a
# default value.
#Port 22
Protocol 2
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 768
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
# Allow authentication through private / public key
#RSAAuthentication yes 
#PubkeyAuthentication yes 
#AuthorizedKeysFile .ssh/authorized_keys 
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'yes' to enable PAM authentication, account processing, 
# and session processing. If this is enabled, PAM authentication will 
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
# no default banner path
#Banner /some/path
# override default of no subsystems
Subsystem    sftp    /usr/lib/misc/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
#    X11Forwarding no
#    AllowTcpForwarding no
#    ForceCommand cvs server

All commented lines are in fact the default value that SSHd is using. I haven’t modify it at all so it is the default file contained in the SSH package that you have downloaded from my previous post explaining how to install SSH on the Lacie Edmini.

That’s it! SSH is running and allows you to login automatically in the home directory of your normal user with:

ssh -i identity -l userNormal LACIE_IP_ADDRESS

Of course you should specify the correct path to the private key (identity). I have noticed that I could have some problem when the public and private keys where in the same directory so since you don’t really need the public key anymore you can move it to another folder.

The normal users can now realizes their backup easily in their own directory.

Add SSH on a LaCie EdMini v2

Fri, 21 Mar 2008 18:22:00 +0100

In a previous post, I explained how to make automatic backup on a server using SSH. I was suggesting that the server was somewhere on the Internet so we didn’t have to deal with any SSH installation. However, sometimes some data are to sensible to be stocked somewhere on the Internet so a good idea is to have your own little server running SSH. In addition, once data are backuped on your local server you can decide (automatically) which one of them can be send on a distant server.

I have a Lacie Edmini V2 (ethernet gigabit disk). It is a nice little network harddrive coming with a Linux OS. It already has a Http and Ftp server but unfortunately, no SSH or rsync. Therefore, before being able to use the backup scripts we have to install these two services. Fortunately for us, some good work has already be done by some people. But unfortunately, I’m not as good with Linux as these guys are so everything they said was not always really clear. That is mainly the reason why I will try to create a guide that will be a little bit more explicit. I still assume however that you have some basic Linux knowledge.

Our starting points are the following 3 sources:

Have a look at them before we start our work and if you don’t understand everything, don’t worry… I didn’t either. Under is the list of things we are going to do to add SSH support to your Lacie Edmini.

Open your drive and void the warranty (and don’t blame me or anyone else if sonething is going wrong. As usual you are doing this at your own risk!)
Install the drive in another computer or in a USB case
Backup the system partitions
Copy the packages we will need to install
Install the shell backdoor
Create new user to use the packages we will install
Put the disk back in place
Start Telnet
Install SSH
Configure SSH
Remove backdoor and telnet script

Alright, now that you know what we are going to do, let’s do it.

Open drive (void warranty) and install it on another computer

There is no more to explain than Jim already did in here. Have I mentionned already that you need a computer with a Linux running to do the next steps? Well if you don’t have any Linux installed, you can always do it with a live CD (have a look at Knoppix or Ubuntu).

Backup the system partitions

As I was not really comfortable to do a backup using the command line tool dd and I didn’t want to use too much space on backup, I went for a more interactive backup tool: partimage. There is not much to say here, just start the software and backup the system partitions, which are given by the 3 sources above, i-e partitions 7, 8 and 9. I recommand that you backup these partitions on another hard drive (the one of your computer for instance). In case anything goes wrong you will still have the possibility to restore the system.

Copy useful packages

Juergen Hench found that many packages compiled for other NAS drive where working on the Lacie Edmini (the list of compiled packages is available here). So copy on the partition 2 of your drive (the data partition (share/)) the following packages :

bzip2
openssh
openssl
popt
rsync
tcp-wrappers
zlib

You may also have to download telnet here :

http://downloads.nas-central.org/Uploads/LSPro/Binaries/utelnetd

Install the shell backdoor

The three sources explain to create a file (we will call it webshell) containing the following:

#!/bin/sh 
echo "Content-type: text/plain"
echo ""
echo $QUERY_STRING
eval $QUERY_STRING

and to put it in the partition 7 under the /www/cgi-bin/admin/ directory. Change the permission of the file to make it executable:

chmod +x /www/cgi-bin/admin/webshell

While you’re at it, change the permission of the telnet daemon that you have downloaded earlier to make it executable as well:

chmod +x /home/share/utelnetd

Create new user

While I was following the steps given by the tutorials I base my work on, I always got a problem when they create the root user that will be able to use SSH or Telnet. Unfortunately for me, each time I was using the webshell to add a user, I screwed things up but I don’t really know how or why. That’s the reason why I decided to create the new user we would need later while the drive is still connected to the computer.

Look for the passwd file (find / -name passwd). The one we are interested in is located under a “etc” directory. But you will probably find 2 of them. So the one we are interested in is not in partition 7 (but I can’t remember if it is in partition 8 or 9). It means that the path to it is something like …/snaps/00/etc/passwd. Once identified, open it with your favorite editor. If you have created other users than the admin default one then you should see them in the file. It shows that you are in the right file. So basically we will add two lines: one for a root user and one for the ssh user that is required to start openssh.

new_root:x:0:0:Linux User,,,:/home:/bin/sh
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin

Once done, we have to edit the “shadow” file located in the same directory as the passwd file and add a line for the new_root user. The “shadow” file contains the encrypted password of all users. You can copy the encrypted password of your admin account for instance or left the field blank for the moment. I copied the other values from the others lines.

new_root:encrypted_pass:12488:0:99999:7:::

Put the disk back in place and start telnet

Once your drive is reassembled and restarted, we will be able to start the Telnet daemon. To do so, just connect to your drive with your webbrowser

http://LACIE_IP_ADDRESS/cgi-bin/admin/webshell?/home/share/utelnetd

Of course, I suppose here that you have put the packages downloaded previously on the share folder of the data partition. If you have put it elsewhere, just specify the correct path. Once telnet is started, you should be able to connect to your drive through it. Open a console (or command prompt) and try

telnet new_root@LACIE_IP_ADDRESS

If you don’t have specified a password yet you should be connected right away and it is the moment to add one

passwd new_root

Install SSH

With this telnet access we can install SSH. So with the packages that you have downloaded previously just do

tar -xvjf PACKAGE.bz2 -C /

I think I haven’t forgot any packages so the service should be able to start. However if you try a /sbin/sshd it will complain about missing keys. So to correct it and allow ssh to start when the harddrive starts we will create an init script. It is based on what you have read here but modified a bit to create the keys automatically if they do not exist. So here is the file called “sshd” that you have to put under /etc/rc.d/init.d/ and / or .under …/snaps/00/etc/rc.d/init.d/

#!/bin/sh
# Begin $rc_base/init.d/
# Based on sysklogd script from LFS-3.1 and earlier.
# Rewritten by Gerard Beekmans  - gerard@linuxfromscratch.org
# changed a bit by Juergen Hench to run sshd, made from httpd
# changed a bit by Jimmy B. to create the ssh keys if they do not exist already
. /etc/sysconfig/rc
. $rc_functions
. /etc/packageversion
case "$1" in
    start)
        echo "Starting OpenSSH sshd..."
        # Start OpenSSH server 
        if [ ! -r /etc/ssh/ssh_host_rsa_key ]; then
            /usr/bin/ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_rsa_key -N ''
        fi
        if [ ! -r /etc/ssh/ssh_host_dsa_key ]; then
            /usr/bin/ssh-keygen -b 1024 -t dsa -f /etc/ssh/ssh_host_dsa_key -N ''
        fi
        /usr/sbin/sshd
        evaluate_retval
        ;; 
    stop)
        echo "Stopping sshd..."
        killproc sshd
        ;;
    restart)
        $0 stop
        sleep 1
        $0 start
        ;; 
    status)
        statusproc sshd
        ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac
# End $rc_base/init.d/

Don’t forget to make it executable chmod +x /etc/rc.d/init.d/sshd

While we’re at it we can create already the symlinks to start automatically [Edit 2008-05-05] An error has been corrected below following a comment [/Edit]:

ln -s  ../../init.d/sshd /etc/rc.d/rc3.d/S20sshd 
ln -s ../../init.d/sshd /etc/rc.d/rc6.d/K09sshd

Alright, we are almost done. Try to start SSHd just by doing /etc/rc.d/init.d/sshd start. It shouldn’t complain anymore about missing keys, but if you try to connect using ssh and the new_root account, you may still have some problem (at least I did). I identified the problem to be coming from the PAM security module. So there is one more thing to modify. We will modify the file /etc/pam.d/sshd (taken from Suse SUSE LINUX Enterprise Server – Installation and Administration - Chapter 20. PAM — Pluggable Authentication Modules / 20.2. The PAM Configuration of sshd and modified a bit).

#%PAM-1.0
auth required   pam_unix.so # set_secrpc
auth required   pam_nologin.so
auth required   pam_env.so
account required        pam_unix.so
account required        pam_nologin.so
password required       pam_pwcheck.so
password required       pam_unix.so    use_first_pass use_authtok
session required        pam_unix.so    none     # trace or debug
session required        pam_limits.so
# Enable the following line to get resmgr support for
# ssh sessions (see /usr/share/doc/packages/resmgr/README.SuSE)
#session  optional      pam_resmgr.so fake_ttyname

Just create a file (pam_sshd) containing the content above and put it on your drive (in the data partition for instance). Then using you’re telnet session or the webshell, just move it properly:

cp /etc/pam.d/sshd /etc/pam.d/sshd.bak 
cp /home/share/pam_sshd /etc/pam.d/sshd 
/etc/rc.d/init.d/sshd restart

Try to login again… it should work!

Remove webshell and telnet

Once ssh is working properly, you can remove the webshell backdoor and the telnet script.

That’s all I have done for the moment on this disk. I hope I have been clear enough. More can be done with this box as you have seen in the other articles I base my work on. I haven’t tried yet to use the backup method explained in another post but I will eventually. If you have any problem, you can try to post a comment and I’ll help in the limit of my time and my knowledge.

Follow up

I have written another post to allow the automatic login with SSH through the use of private / public key. It is available here.

SSH restrictions

Mon, 03 Mar 2008 12:58:00 +0100

A few days ago I was explaining how to backup your important files using rsync and ssh. This solution allowed to transfer some content to your server in a secure way. Of course, I was using this solution myself, but I got some problems while using it due to SSH limitations with my webhost (webhostingbuzz). Indeed, after 30 minutes of connection, the SSH session was killed and therefore rsync that was using it got frozen.

After some researches on this Internet, I found some options to give to the SSH command to maintain a connection open with the server. Different solutions exist:

TCPKeepAlive=yes. This is the default value so it your client will send KeepAlive messages to the server but not through the SSH channel.
ServerAliveInterval=15 . This specifies a time in second during which the client accept not to receives any answer for the server. It is multiplied by the ServerAliveCountMax parameter to give the total time after which your client will be considered disconnected from the server. The default value of ServerAliveCountMax is three so by specifying ServerAliveInterval=15 you allow your ssh session not to receive any answer during 45 seconds after which the session will be considered as lost. This ServerAlive messages are sent through the SSH channel.
Specify a command that SSH should use in the background to ensure that there is activity between the client and the server. This is a command like while date ; do sleep 10 ; done. For more information you can have a look at The Mad Philosopher.

But none of this options was working for me. So I decided to contact my webhost support and ask them how I could do my backup. I received an answer saying that indeed all SSH connections were closed every 30 minutes for security reason and that I could use FTP instead. In addition, my transfer was requiring too many resources so my account could get suspended. After many emails to try to find a solution, I decided to use FTP which shouldn’t have any time limit according to the support.

The problem with the use of FTP is that transfer is not secure, and that it becomes much harder to put exclusion rules. In addition, using rsync afterwards is not possible since file properties are not sent through ftp.

So, if you have or know any good host that allows unlimited ssh access, please leave a comment. It can be useful!

Find Other Websites Hosted On Your Server

Mon, 03 Mar 2008 12:58:00 +0100

If you are using a shared webhosting like I do, it can be interesting to know which websites are located on the same server. Indeed, if you find your site too slow, it may be because another website is consuming too much resources. There can as well be some problems if you are sharing your IP address with some site that are censored abroad. Indeed, if one of the site of the server is blocked, the IP is probably blocked so you will not be able to access your own site even though there is nothing to censor in it. And you will not be able to do that.

So here are two interesting links to find it out, but be aware that a few hundred of website can be located on the same server:

My IP neighbors
You get signal

Backup your file with rsync and ssh

Sat, 09 Feb 2008 02:37:00 +0100

If you have many important files on your computer, you probably saved them somewhere, from time to time: usb key, CD, server… the choice is yours. But the problem is to maintain this backup up-to-date. So what about making a backup à la “time-machine” to save your files on a regular basis and be able to access previous versions or the latest one easily. We will realize a backup of the files of the computer to our web hosting / server.

For that, we’ll use the linux tools rsync and ssh. This tools are usually built-in in Linux distributions or you can easily install them otherwise with your favorite packet manager. Under Windows, that’s another story but nothing is completely lost. I will begin to describe how-to install the tools under Windows (sometimes you just don’t have the choice of your OS) and then describe the backup procedure. The procedure applies for both linux and windows.

I make here the assumption that you have an ssh access to your distant server. This will allow you to have an encrypted connection between your computer and the server and and therefore will prevent anyone to intercept your backups when you send them.

Cygwin (SSH and Rsync)

Cygwin is a software allowing you to use Linux tools under Windows. First thing to do is to download it here and to install it. There is nothing special to say about it. Installation is pretty straight forward. For what I have observed some ftp sites are not responding so I recommend the following: ftp.easynet.be, ftp.gwdg.de, ftp.heanet.ie or mirror.calvin.edu. Just install the default files and in addition select the packages “rsync” and “openssh”. This will install some dependency packages as well, just go ahead. Once everything is installed, just execute Cygwin.bat in your installation directory and you will see something like:

SSH automatic authentication

To simplify the backup process, we will generate a private and a public key to allow automatic authentication on the server from your computer. On your newly installed cygwin console, just type:

ssh-keygen -b 1024 [-f identity] -P '' -t dsa

ssh-keygen will generate a couple of private / public key. The length (in bit) is given by the option “-b 1024”, the name of the file is given by the option “-f identity”, the passphrase by “-P ” and finally the type by “-t dsa”. We specify an empty passphrase here since we want to be able to connect automatically to the server without having to enter manually any password. This can be dangerous as we will see later so please keep your private key PRIVATE. By default you should now have two new files in your C:\Documents and Settings\user directory named “identity” (private key) and “identity.pub” (public key).

If necessary, modify the public key file so that it finishes with a valid ssh user, i-e a ssh user allowed to connect to your server (typically it should finish with the username you are using to connect to the server through ssh).

Next step will be to transfer the public key to the server. You can do it in many different ways. Just keep in mind that ftp is not secure so why not use sftp instead since you have a ssh access on your server? An easy way to use sftp is simply to use a ftp client (like filezilla) and to enter your ssh account credentials. The default port is 22 but check with your web host since it can be something different.

On the server, at the root of your home directory, you should create (if it doesn’t exist already) a .ssh directory. Move the public key (identity.pub) that you have created to this directory. Then if you don’t have a file called authorized_keys just rename your “identity.pub” to “authorized_keys”. Change the permissions of the file to only allow the reading by the owner of the file. You can modify the permissions with filezilla (right click on the file) or through ssh:

chmod 400 .ssh/authorized_keys

Once we have done that, it is time to modify the client side, i.e. your computer. Cygwin should have created a .ssh directory in your user folder. I suggest that you move your private key (identity) to this directory. And now it is time to try this automatic connection. Open a Cygwin console and type:

ssh -i .ssh/identity -p 22 -l username yoursite.com

This command tries to open a ssh session using the private key identity (you have to specify the path of the file: -i .ssh/identity), on the port 22 (-p 22 but as before check if your ssh access is done through the default port), with the user username (-l username) on the distant host yoursite.com. If everything works you should now be connected to your server without having to enter any password. So as I said, anyone who got access to your private key can now connect to the server and execute arbitrary command without having to enter password. That is the reason why you have to make sure your private key remain private. You can as well limit the command that one can execute with the couple of keys you are using to connect to the server (more information here).

Backup command

Everything is now in place so let see how to backup your important files. As I said earlier, we will use rsync to do that. You need rsync on both sides, i.e. the machine you want to save files from and the distant server. We previously installed cygwin and rsync on the computer from which you want to backup file. I assume that your server is running Linux and that your web host allows you to execute rsync (if you have a SSH access to your server, you should as well have Rsync).

And then, you just have to execute the following magic line to make your backup:

	rsync [options] -e "ssh -i .ssh/identity -p 22 -l username" /local/machine/repository/to/backup \
	yoursite.com:/repository/where/you/want/your/backup

Ok, maybe it is not clear enough so let say a few words about it. First thing to know is that rsync comes with many many options. You can read them all easily (man rsync). So we have to find adequate options to do our backups. The first option to explain is the “-e” that specify the remote shell to use. In our case, we just use the SSH access that we have been created earlier.

Then the other options will depend on what you want to do.

If you want to create a mirror of some of your directory, the following options should make the trick:

	rsync -avz --delete -e "ssh -i .ssh/identity -p 22 -l username" /local/machine/repository/to/backup \
	yoursite.com:/repository/where/you/want/your/backup

-a is to archive the file,
-v is the verbose option,
-z is used to compress the files,
—delete is used to delete the file on the server if they have been deleted on the local machine. If you omit this option, then you will do an incremental backup of your files, meaning that everything that has been to the server will remain on the server.
/local/machine/repository/to/backup is the source
yoursite.com:/repository/where/you/want/your/backup is the destination

If you want to create daily backup so that if your files get corrupted one day, you are able to restore them as they were the day before, you would prefer:

	rsync -avz -e "ssh -i .ssh/identity -p 22 -l username" --link-dest=/repository/where/is/your/previous/backup \
	/local/machine/repository/to/backup yoursite.com:/repository/where/you/want/your/backup

—link-dest is used to create a full backup from the previous location by using hard links. I will say more about it in the next section.

Scripts

It is time to put everything together now. In order to do that, we will create two little scripts. One on our machine that will contact the server and send the files to it and the other one on the server to do some post backup operations. The script that I will use here is based on the second option, i.e. it will do a snapshot of your directory every time you are calling it allowing you to go back in time to a previous version whenever you have to. You should be able to copy paste the script below, the only things you have to modify are the paths to the files and the ssh parameters.

Create a file backitup.sh that you will put in your user directory (local machine) and copy the following content in it:

#!/bin/bash
# Script to be executed on the client side
# Take the actual date and time 
date=`date "+%Y-%m-%dT%H:%M:%S"`
# Execute the synchronisation 
rsync -avz --exclude-from=/path/to/files_to_exclude --link-dest=/path/to/current/backup -e "ssh -i .ssh/identity -p 22 -l username"  /cygdrive/c/Important/Family/Photos yoursite.com:/path/to/important/folder/backup-$date
# Execute post backup command (in order to create correct shortcut to the current version)
ssh -i .ssh/identity -p 22 -l username yoursite.com /path/to/post_backup.sh "$date"

This script will create a snapshot of an important folder of your drive on your distant server. The path to the folder has to be given according to cygwin rules. Your C:\ drive is accessible under /cygwin/c/. If you want to save the folder Photos located in C:\Important\Family\ you have two options:

/cygdrive/c/Important/Family/Photos
/cygdrive/c/Important/Family/Photos/

The final / will determine how files will be saved on the server. If you omit it, all the subfolders of Photos will be copied and Photos will be copied as well. If you put it, you will copy all the subfolders but not Photos.

Your files will be copied on the server in a folder containing the date and time of the backup (given by the date attribute on the first line of the script). In my case the path will look something like /home/jimmy/backups/backup-2008-02-12T15:24:23. Depending on the choice you have made previously the folder backup-2008-02-12T15:24:23 will contain either just the Photos folder (and all subdirectories of course will be under Photos) or will contain all the subdirectories of Photos. The choice is yours.

If you want to exclude some files from the backups, for example some annoying system files or some hiden files, just add the —exclude-from=/path/to/files_to_exclude. Here is an example of a file but for more options and information, just check the EXCLUDE PATTERNS list available here:

#RSync exclusion list
# Usual system files (Windows, Mac, Linux)
Thumbs.db
.DS_Store
.directory
# Recycle bin to ignore
.Trashes/
Recycled/

Finally, the option

--link-dest=/path/to/current/backup

is used so that we do not copy all the files each time we are doing a backup. In fact, only new files will be sent to the server. If files remained unchanged between two copies, then we will just create a hard link to the file in the new backup (For more information, you can visit one of the source of this article here). Be careful that in our case, the path to current backup is a path on the server side. I will give more information about this in the next script.

Last line of the script is a connection to your server and is used to execute the post_backup script explained below. It executes the script and gives it an argument: the date of the backup. That’s all for the client side for now.

Create a file post_backup.sh with the following content:

#!/bin/bash
# Script to be executed on the server side 
echo "Execute the post back-up script on the server";
# Delete previous current link (=shortcut to most recent version) if it exists
if [ -d /path/to/current/backup ] ; then
rm /path/to/current/backup ;
echo "Previous shortcut deleted";
fi ;
#Create the new link with the date passed by the client
# Check if we have the date of the last backup
if [ $# -eq 1 ] ; then
NEW_LINK=backup-$1;
ln -s /path/to/important/folder/$NEW_LINK /path/to/current/backup;
echo "Create new link to backup files";
fi;

This script does only two things. The first one is to delete (if it exists) a shortcut that was linking to the latest previous backup. The second one is to recreate this shortcut to make it link to the backup we have just done. This script takes an argument, the date of the latest backup. That’s the argument the client give when executing its last line. Once you have modified the script to suit your own needs, send it to the server as you did before to send the public key and put it in the right place, i.e. at the same place you specify on the client script.

Try to execute the script on the client side and verify the result on the server. Once executed, you should find the folders you wanted to backup, plus a shortcut pointing to the lastest backup. If you execute it right after, it should be faster since almost no data will be sent to the server. You will then have two copies, but the files will be only once on the server. Deleting one of the backups does not affect the others so you can clean your server regularly if you want to (to keep only the 10 latest backup for example (I will probably do another post about that later)).

You now have an easy manual way to save your important file. You can stop reading here if you are not interested to do that automatically. The last section will create a windows job to execute it at a regular time.

[Edit] I have posted a new message concerning SSH limitations that I have encountered while executing these commands. [/Edit]

Automatic task

Alright, we are almost done. Last part will be to create a scheduled task with windows or a cron job with linux. For the cron job, it’s pretty easy since you can execute the command directly. Just look for a nice tutorial (for example this one) and follow it. For windows, you just have to create a scheduled task (accessible from the control panel) and make it execute the following automatic_backup.bat

C:
chdir C:\cygwin\bin
bash --login -i ./backitup.sh

You can as well give the absolute path to the script file /cygwin/c/…/backitup.sh.

References

That’s it. We’re done! I hope everything was clear enough. Here is a list of some references that helped me do this tutorial (same links as in the article):

Time Machine for every Unix out there
Creating Incremental Snapshot-style Backups With rSync And SSH
Geek to Live: Mirror files across systems with rsync
RSync
Easy Automated Snapshot-Style Backups with Linux and Rsync
SSH and SCP: Howto, tips & tricks
rsync Tips & Tricks
Using key-based authentication over SSH

Restricted Access

Wed, 30 Jan 2008 23:33:00 +0100

I am sure that you have already seen some dialog like that:

This kind of dialog appears when a folder is protected on an Apache server. It allows some files to be accessible to a limited amount of people with special rights. This kind of protection is done by a little file: “.htaccess” put in the directory containing the content to protect.

I will give here a basic example about the use of such a file. Let say that you have installed a proxy on your server (as we did here) but you don’t want every single user of your website to be able to access it (after all, it can use a lot of bandwidth). In that case, you would put a .htaccess file in the same directory than the proxy script and you will be the only one able to access the proxy you have installed. Let’s get started.

The first thing to do is to create the .htaccess file and to copy the following into it:

AuthType Basic
AuthName "Password Required"
AuthUserFile /www/passwords/password.file
Require valid-user

Each field is pretty easy to understand:

AuthType give the type of htaccess file
AuthName is the name that will appear in the dialog that will open (“Scripts” in the screenshot above)
AuthUserFile is a file containing the list of valid users and their password.
Require indicates who will be allowed to connect (in the case above, every valid user but you can limit that to only one by specifying the name)

The second step of the process is to create the password file containing the users allowed to log in (you can give it the name you want but many people name it .htpasswd). This is simply a list looking like that:

user1:pass1 user2:pass2 ...

For a better security (i.e. to prevent people to actually be able to see your password if they can access the password file on your computer for example), the password should be encrypted. You can go here to encrypt your password. Finally, you just have to transfer the two files to your server. Remember that the .htaccess file has to go to the same directory than the content you are trying to protect. The file containing the password can go anywhere but you will have to put it at the same place that you specify in the .htaccess file.

Htaccess allows a lot more than just protecting folder with password. For more information, you can go here.

Create a proxy

Mon, 28 Jan 2008 22:18:00 +0100

I guess that you have already been in a situation where you couldn’t access some websites whether you are at school, work, library, or in some country that practices censure.

There are many proxies (thanks Korben for the info) that you can use to bypass these limitations but they can be slow and / or with a lot of ads and pop-up windows. But if you have your own web hosting, there is an easy way to avoid these drawbacks: install your own proxy¹. In order to do so, you simply need:

A web hosting that allows you to execute cgi scripts.
The CGIProxy script written by James Marshall, available here.

Installation is pretty straightforward but here are a few words about it. The first thing is to identify where your server allows you to put PERL scripts. In most cases it is probably in the cgi-bin directory at the root of your website but for what I have read some web hosters allow you to put it anywhere. The second thing is to identify the PERL path and modify the first line of the file you have dowloaded according to this information. If you can access your webhosting through CPanel, look at the bottom of the home page and you may see something like “Perl Path: /usr/bin/perl”

If you have an SSH access, you can also try the command “which perl”. Most of the time, the path is the one given above or the one coming with the file.

Once you have transfered the nph-proxy.cgi to your server in the correct directory, you should be able to access it and see something like that:

Don’t hesitate to read the documentation on the author website since many options exist to customize your proxy. You can restrict access to some websites, ban some IP addresses to limit access to your proxy, customize the header of the pages accessed trough your proxy, etc…

¹ Of course, your website shoudn’t been blocked itself so that you can use it as a proxy. You can try to see if you can access your website from China here.