Windows command prompt vs PowerShell vs Cygwin for remote backup scripts

I’ve been struggling with trying to get a new backup routine working for my Laptop.

I should point out that I have several complex requirements for backup so my needs are probably not average.

However, it really shouldn’t be this hard!

I need to use a combination of BZR (Bazaar) for document version control and RSYNC (for files that don’t need version control and for those folders that might contain files too big for version control systems – around 1/3 to 1/2 available memory).

All of the backups need to happen over a secure link since I am often outside of my home network – indeed quite often behind locked-down enterprise firewalls but that’s a story for another time. So I use SSH (Secure SHell) to manage the secure connection and transmission. Thankfully both BZR and RSYNC can both use SSH as a transport.

I don’t want to have to enter my remote system password loads of times though and this is where things started to get annoying. Using the Windows native versions of BZR, RSYNC and SSH I could not get a single shared password to work no matter what I tried.

I also had some problems trying to control the output from the various tools and use it to further control what happens next – for example getting an IP address and working out whether I am on a network and where that network is.

I tried to do this with a Windows command script first but even with the Windows 7 extensions it really is far to hard to get anything useful done and when I found myself turning to more and more utilities to help I thought “enough is enough”.

At that point I happened to be reading an article on Windows PowerShell, the .NET scripting host so I thought I’d give it another go (having tried it before). I soon found that, although powerful for controlling the WMI interface, it is desperately convoluted and annoying for general use.

So, realising that most of the tools I wanted to use have their roots in the UNIX world, it would make sense to try out the latest version of Cygwin. This has really come on a long way since it’s early days and is far more mature. It is also very much lighter in weight than the Microsoft provided UNIX services for Windows or whatever they are currently calling it. The Microsoft provided tools load perhaps a dozen services into memory permanently though they are rarely required. Cygwin only uses memory when it needs to.

After converting my backup script from Windows batch to a BASH script under Cygwin, I soon had everything working as I wanted it – including the seemingly intractable problem of the shared passwords, now using KEYCHAIN to manage the SSH-AGENT and keys. So now I only need to supply a password once, it is held reasonably securely in memory and used by SSH as and when required. I only need to enter it once per reboot.

The full script not only backs things up, it also auto-commits changes to BZR and changes network settings to match my current location and proxy server requirements. The later is not yet converted from batch as I don’t need it just at the moment.

Let me know if you are interested in a copy of the script and I’ll upload it somewhere.

Keeping Control: File and Folder Links for Windows Users

A good backup strategy for any computer involves keeping control of where stuff is stored. The fewer locations that contain files that change, the fewer locations have to be maintained.

UNIX users have always had the ability to keep things wherever they wanted and then to LINK that information into the required location. Basically, links create a link or tunnel between one file or folder and another. Most of the time, you will not notice that you’ve entered a tunnel and you are not interested really.

Windows users, however, have always been the poor cousins here. Stuck as we were in FATland, we had no access to fancy features such as links. So Microsoft in their inimitable fashion created a poor-man’s link – the Windows Shell Shortcut – so that the Windows GUI had some minimal capability (really only for menu’s and Windows Explorer).

Windows 2000 improved on this by introducing “Reparse Points” one form of which is the “Junction“. This is an extension to NTFS that allows folders to be joined (linked) to another location in the local volume space. Making junctions is not an obvious process, you can do it from the disk manager and there is a tool in the Windows 2000 server resource kit called linkd. The POSIX tools included in the resource kit contain the UNIX command ln which can also create junction points and hard links; fsutil in XP can also. There are some third party tools too.

It’s odd because I seem to remember that OS/2 had some kind of linking feature.

Anyway, links of the UNIX type are a massively useful feature that has finally (with Vista, Windows 2008 and beyond) made it fully to NTFS and Windows.

Vista, Windows 2008 and Windows 7 all have a command line tool called mklink. This can be used like the Unix ln command to create both hard links (which must be on the same volume) and soft links. Soft links under Windows can, in fact, span across SMB network drives as well.

You might also like to look at another free tool called “Link Shell Extension” by Hermann Schinagl. This integrates into Windows Explorer, the web site also has a more complete explanation of the history of links in Windows. LSE does a number of clever things and is well worth a look. Hermann also has a “dupmerge” tool on his web site that will replace duplicated files with hard links.

So now, if we want to tweak the HOSTS file for example (c:\windows\system32\drivers\etc\hosts), we don’t need to leave in place since that would mean that we would need an extra backup routine. Instead, copy it to somewhere that already gets backed up. Delete the original file and then from the command line:

mklink c:\windows\system32\drivers\etc\hosts %USERPROFILE%\BACKUPS\hosts

Now you can edit the hosts file from either location, there is only one file (in %USERPROFILE%\BACKUPS). The difference being that even if you delete the file from its normal location, it will still exist in the “real” location. If you delete it from its “real” place in BACKUPS of course, the link will be broken and wont work.

To link a complete folder, it is the same command with a /D parameter added. For example, I keep a folder of command line utilities such as ls, ssh and rsync in a folder on a USB pen drive. I sync that folder to the BACKUPS location on my hard drive for convenience but I need the folder in my PATH otherwise its hard to execute the utilities. I don’t want a really long path, it’s bad enough already, so I link the folder to c:\cmd with the following:

mklink /D c:\cmd %USERPROFILE%\BACKUPS\PEN\cmd

Now I add c:\cmd to the path and the utilities seem to be in both places.

I’ve said in other posts that I like to reinstall Windows now and again but it can be a pain to restore all of the document files. Similarly, if you keep multiple operating systems on your hard drive, how do you keep your documents sorted? One way is to put all documents, videos, music, etc. onto a separate partition. Now, instead of going mad with the Windows registry trying to relocate your normal documents folders to another drive. Simply delete the normal documents folder – %USERPROFILE%\Documents\ under Windows 7 and relink it to the appropriate folder on the other drive as so:

mklink /D %USERPROFILE%\Documents d:\Docs

Put this in a script that you run when you reinstall Windows and its easy and quick.

One final note. You may find a few pieces of software that cannot cope with links. Certainly Subversion cannot though Bazaar can. Windows Explorer seems OK though as do utilities such as RSYNC.


Technorati : , , , , , ,
Diigo Tag Search : , , , , , ,

Version Control for Mortals

Version control systems (VCS, or Revision Control Systems or Source Control Systems) are designed for software developers. They enable one or more people to work on source code, annotate changes, split and merge the code, link to bug tracking systems and a number of other things that are interesting to developers but not to most people!

Indeed then, for most people, you might expect that version control systems are not interesting at all. But you’d be wrong.

What makes them interesting to most people is the fact that most of us are very poor at looking after those all-important files that make up our business assets and often personal assets too. We copy, save and delete stuff without giving too much thought to what we are doing. Then later on we scratch our heads and wonder what happened to xyz piece of critical information. I wonder how many times you’ve gone through your emails to get a document back that you know should be (and may well be) on your hard drive somewhere. With desktop search systems now all the rage, you will probably find the document but then you realised that you used it as a template for another document and accidentally saved over the top! Or it got deleted when you were tidying up the old project folders …

Well, in step the version control systems to save the day. They will benefit anyone who recognise the above scenarios.

I’ve been using a VCS for over a year now for my day-do-day documents. I’m glad I did too as I used it to recover most of my documents after a drive failure earlier this year.

SVN LogoI started by using Subversion (SVN). This is a Centralised Versioning System. It requires a central server that is the hub and master for all documents and changes. It is very well supported and many low-cost web hosts also provide Subversion servers.

It is not ideal though for managing general documents. Firstly it does get quite slow (especially for larger files) and changes can only be committed over a live network connection so it’s no good for disconnected work. Secondly, I found it very sensitive to how it was used and I’ve often managed to get my repository in a mess that was very hard and very time consuming to recover from. This is not acceptable in a system that you have to rely on. I’m sure its fine for its original purpose of source control but it is not so good for managing day-to-day work.

Hg LogoNext I looked at Distributed Version Control Systems. The most popular of these (the free ones anyway) are: Git, Mercurial and Bazaar. Mercurial seemed to be the one best developed for Windows so I tried that. It does seem reasonable but it seems to balk quite a bit at large’ish files (a few Mb, it seems that the Windows interface at least hangs quite often at least on Windows 7) and that makes it unsuitable for our needs. Git seems to have a lot going for it but is not so well developed for Windows and is rather more complex, I haven’t tried it at least yet. So that leaves Bazaar. I’m now using Bazaar in anger and I’m quite impressed. It seems to handle large files sensibly, it is easy to set up (really easy), it doesn’t complain when you move files and folders round, it doesn’t get in the way, it’s reasonably fast. You can also use it with a central repository too like SVN.

Bzr LogoBazaar comes with integration to Windows Explorer but you will probably want to look at the command line options too for automation.

I’ve set up a schedule that runs a commit of my main repository “workdocs” every morning, noon and afternoon (9am, 12pm and 4:30pm) and the extra data that is kept in the repository is simply backed up as part of the regular backup since it is just a hidden folder in the root of the “workdocs” folder.

I then commit changes manually as and when I want to after making significant changes to files.

For really critical files, you could couple Bazaar with the file change detection of SyncBackSE to automatically commit changes or you could use AutoHotKey to intercept the <ctrl>-s key combination to run a commit before or after doing a save.


Technorati : , , ,
Diigo Tag Search : , , ,

Shell script to Back up critical files (using RSYNC)

Following up from my article on backing up USB drives, this recipe backs up the critical files on my desktop to remote storage (a NAS device on my network). Note that PC2 is the desktop to be backed up, SLUG1 (192.168.1.2) is the NAS device and USER1 is the user id doing the backup.

#!/bin/bash

# Backup Key PC2 files to Slug1

# Sync 2007 picture folders
##rsync -rl /home/user1/pictures/2007/  [email protected]:/public/pictures/2007/

# Ensure that /mnt/slug1-root/ is mounted
#if [ ! -e /mnt/slug1-root/user1/backups/PC2/bin/ ]; then
#    mount-slug-root.sh
#fi
# Ensure that /media/slug1-public/ is mounted
#if [ ! -e /media/slug1-public/DISK1.txt ]; then
#    mount-slug-public.sh
#fi

# NOTE that to configure the rsync sessions on SLUG1, edit the file /opt/etc/rsyncd.conf
[email protected]::public'
# Use this form if not using sessions
[email protected]:/public'
# Or use this form if the remote folder is mounted locally
#TOPUB='/media/slug1-public'
[email protected]::pc2'
[email protected]:/user1/backups/PC2'
#TOJK='/mnt/slug1-root/user1/backups/PC2'

JKDT=`date --rfc-3339=date`
JKLOG="/home/user1/Backups/pc2backup_$JKDT.log"

echo "Starting PC2 backup at `date`" >$JKLOG
echo "=================================================================="
echo "Starting PC2 backup at `date`"
echo "The log file is at $JKLOG, all backups are to SLUG1/pc2 or SLUG1/public"
echo " "

#--out-format=FORMAT     output updates using the specified FORMAT
#--log-file=FILE         log what we're doing to the specified FILE
#--chmod=CHMOD
#--exclude=PATTERN       exclude files matching PATTERN
#     --exclude-from=FILE     read exclude patterns from FILE
#     --include=PATTERN       don't exclude files matching PATTERN
#     --include-from=FILE
#--dry-run
#OPTS='--verbose --archive --recursive --links --perms --executability --owner --group --devices --specials --times --human-readable --delete --delete-after --stats --ipv4 --progress --password-file=/home/user1/bin/tmppw.tmp --dry-run'
OPTS='--verbose --archive --recursive --links --executability --devices --specials --times --human-readable --delete --delete-after --stats --ipv4 --progress'
echo "Back up various bits - WARNING: DELETES files from destination" >>$JKLOG

RSYNC_PASSWORD=`kdialog --password "Password for [email protected] please:"`
#kdialog --password "Password for [email protected] please:" >~/tmppw.tmp

echo "Backups to SLUG1/pc2"
echo " "
# ** JK BACKUPS **
echo "user1/bin"
echo "rsync $OPTS /home/user1/bin/ $TOJK/bin/" >>$JKLOG
rsync $OPTS /home/user1/bin/ $TOJK/bin/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG
echo "user1/backups"
echo "rsync $OPTS /home/user1/Backups/ $TOJK/Backups/" >>$JKLOG
rsync $OPTS /home/user1/Backups/ $TOJK/Backups/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG
#echo "rsync $OPTS /home/user1/Downloads/ $TOJK/Downloads/" >>$JKLOG
#rsync $OPTS /home/user1/Downloads/ $TOJK/Downloads/ >>$JKLOG 2>&amp;1
#echo "=========================================" >>$JKLOG

echo "Backups to SLUG1/public"
echo " "
# ** Backups to public **

echo "user1/ebooks"
echo "rsync $OPTS /home/user1/eBooks/ $TOPUB/ebooks/sorting/" >>$JKLOG
rsync $OPTS /home/user1/eBooks/ $TOPUB/ebooks/sorting/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG

echo "user1/pictures/Lnnnn"
echo "Back up picture files - WARNING: Does NOT delete files from destination" >>$JKLOG
OPTS='--verbose --archive --recursive --links --times --human-readable --stats --ipv4'
echo "rsync $OPTS /home/user1/Pictures/L2007/ $TOPUB/pictures/2007/" >>$JKLOG
rsync $OPTS /home/user1/Pictures/L2007/ $TOPUB/pictures/2007/ >>$JKLOG 2>&amp;1
echo "rsync $OPTS /home/user1/Pictures/L2008/ $TOPUB/pictures/2008/" >>$JKLOG
rsync $OPTS /home/user1/Pictures/L2008/ $TOPUB/pictures/2008/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG

echo "user1/backups/usbpen1 &amp; usbpen2"
echo "rsync $OPTS /home/user1/Backups/USBPEN1/ $TOJK/Backups/USBPEN1/" >>$JKLOG
rsync $OPTS /home/user1/Backups/USBPEN1/ $TOJK/Backups/USBPEN1/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG
echo "rsync $OPTS /home/user1/Backups/USBPEN2/ $TOJK/Backups/USBPEN2/" >>$JKLOG
rsync $OPTS /home/user1/Backups/USBPEN2/ $TOJK/Backups/USBPEN2/ >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG

echo "Google Earth places"
echo "Back up Google Earth myplaces.kml" >>$JKLOG
rsync $OPTS /home/user1/.googleearth/myplaces.kml $TOPUB/maps+walks/pc2-myplaces.kml >>$JKLOG 2>&amp;1
echo "=========================================" >>$JKLOG

#echo '' >~/tmppw.tmp
#rm ~/tmppw.tmp

echo " "
echo "ENDING PC2 backup at `date`" >>$JKLOG
echo "ENDING PC2 backup at `date`"
echo "=================================================================="

# To run under schedule
#    Log
#    Replace password

I have a similar script that runs on the NAS device which backs key files on that to a remote hosting service on a different continent! That way, I don’t have to worry about the house burning down or being burgled.

Automatically Backing up a USB Drive with RSYNC (KDE)

USB Drives of all kinds need to be backed up and the best backup is an automatic one (it’s the only way to make sure that it gets done!).

So here is one recipe for doing just that using RSYNC and some BASH scripting magic.

I’ve split this into two files. You don’t have to do this of course and one may well be better for you. I used two because I can run the second one manually as well. Put everything in autorun.sh if you want to backup each drive individually, however, note that KDE produces an annoying extra dialog (a security warning) asking if you really want to run the autorun.

  • autorun.sh
    This resides in the root of the USB drive and is executed automatically by KDE when the drive is detected (though not if the drive is attached when booting)
  • usb-backup-manual.sh
    This is a bit of a nasty hack, I have manually configured a list of drives that might be attached so that I can back them all up together. Not elegant but it works for me.

autorun.sh

#!/bin/bash

# KDE will automatically run an executable file called: .autorun, autorun or autorun.sh (in that order)
# Alternatively, a non-executable file called .autoopen or autoopen can contain a file name
# of a non-executable file on the media which will be opened with the default app for that file.
# See: http://standards.freedesktop.org/autostart-spec/autostart-spec-0.5.html#mounting

# Also see: http://b50.roxor.pl/~michal/linux/autorun.txt
# for some interesting ideas

# Where are we running from? e.g. /media/usbpen1
mediaDir=$(echo $0|sed 's/autorun//')

kdialog --title "USB Drive Backup" --yesno "I'd like to backup the USB drives, can I?"
if [ $? = 0 ]; then
echo " OK Selected, I'm going"
echo "Autobackup run: `date`" >usb-linux-auto-backup.log
exec ~/bin/usb-backup-manual.sh
else
echo " Cancel selected, so do nothing - bye."
fi
</code></pre><h4>usb-backup-manual.sh</h4><pre><code>
#! /bin/bash

#http://www.sanitarium.net/golug/rsync_backups.html
#http://www.mikerubel.org/computers/rsync_snapshots/
#http://rsync.samba.org/examples.html

echo "Starting USB Backup: `date`"
echo "Starting USB Backup: `date`" >~/Backups/usb-backup-manual.log

# From
MNT="/media"
# To
TO="/home/julian/Backups"

dcopRef=`kdialog --progressbar "Starting backup - press cancel to stop further processing (no next step)" 4`
dcop $dcopRef showCancelButton true

#until test "true" == `dcop $dcopRef wasCancelled`; do
for f in "CF2G1" "SD1G1" "USBPEN1" "USBPEN2"
do
dcop $dcopRef setLabel "Backing up $MNT/$f  ==>  $TO"
echo "--------------------------------------"
echo "$f  ==>  $TO"
inc=$((`dcop $dcopRef progress` + 1))
sleep 2
if [ -e $MNT/$f ]; then
  dcop $dcopRef setProgress $inc
  RSCMD="rsync --recursive --times --delete-during --stats --human-readable -h $MNT/$f $TO"
  echo $RSCMD
  echo $RSCMD  >>~/Backups/usb-backup-manual.log
  $RSCMD
  dcop $dcopRef setLabel "RSYNC for $f finished"
else
  dcop $dcopRef setProgress $inc
  dcop $dcopRef setLabel "$MNT/$f not mounted"
  echo "$MNT/$f not mounted"
  echo "$MNT/$f not mounted"  >>~/Backups/usb-backup-manual.log
fi
echo "======================================="
sleep 2
done

dcop $dcopRef close

echo "End: `date`"
echo "End: `date`" >>~/Backups/usb-backup-manual.log

Note the use of KDialog to provide a minimal GUI. In the second file, KDialog produces a progress bar.

Also note the RSYNC parameters. These are always painful to get to grips with so it is nice to have an example to work from. In this case I am backing up so I am making sure that the backup is an exact copy of the original (as opposed to synchronising which would allow changes to happen on either side).