Windows command prompt vs PowerShell vs Cygwin for remote backup scripts

I’ve been struggling with trying to get a new backup routine working for my Laptop.

I should point out that I have several complex requirements for backup so my needs are probably not average.

However, it really shouldn’t be this hard!

I need to use a combination of BZR (Bazaar) for document version control and RSYNC (for files that don’t need version control and for those folders that might contain files too big for version control systems – around 1/3 to 1/2 available memory).

All of the backups need to happen over a secure link since I am often outside of my home network – indeed quite often behind locked-down enterprise firewalls but that’s a story for another time. So I use SSH (Secure SHell) to manage the secure connection and transmission. Thankfully both BZR and RSYNC can both use SSH as a transport.

I don’t want to have to enter my remote system password loads of times though and this is where things started to get annoying. Using the Windows native versions of BZR, RSYNC and SSH I could not get a single shared password to work no matter what I tried.

I also had some problems trying to control the output from the various tools and use it to further control what happens next – for example getting an IP address and working out whether I am on a network and where that network is.

I tried to do this with a Windows command script first but even with the Windows 7 extensions it really is far to hard to get anything useful done and when I found myself turning to more and more utilities to help I thought “enough is enough”.

At that point I happened to be reading an article on Windows PowerShell, the .NET scripting host so I thought I’d give it another go (having tried it before). I soon found that, although powerful for controlling the WMI interface, it is desperately convoluted and annoying for general use.

So, realising that most of the tools I wanted to use have their roots in the UNIX world, it would make sense to try out the latest version of Cygwin. This has really come on a long way since it’s early days and is far more mature. It is also very much lighter in weight than the Microsoft provided UNIX services for Windows or whatever they are currently calling it. The Microsoft provided tools load perhaps a dozen services into memory permanently though they are rarely required. Cygwin only uses memory when it needs to.

After converting my backup script from Windows batch to a BASH script under Cygwin, I soon had everything working as I wanted it – including the seemingly intractable problem of the shared passwords, now using KEYCHAIN to manage the SSH-AGENT and keys. So now I only need to supply a password once, it is held reasonably securely in memory and used by SSH as and when required. I only need to enter it once per reboot.

The full script not only backs things up, it also auto-commits changes to BZR and changes network settings to match my current location and proxy server requirements. The later is not yet converted from batch as I don’t need it just at the moment.

Let me know if you are interested in a copy of the script and I’ll upload it somewhere.

Running Komodo Edit Open Source Code Editor Under Windows 7

The open source version of Komodo’s code editor and development environment Komodo Edit is a great tool for development. I use it for PHP, HTML, JavaScript and more.

However, I haven’t done any serious coding for a while so I haven’t needed to run it under Windows 7 even though I had it installed. When I did, I was disapointed to find it behaving very poorly. It wouldn’t resize properly without messing up the screen. I tried with some of the compatibility settings that Windows 7 gives you but that made no difference. I also tried an upgrade to the latest version.

A quick search didn’t reveal anything about Windows 7 specifically but I did spot a discussion about problems under Vista that were related to file permissions. Sure enough, making Komodo Edit run as Administrator fixed the issues.

I’ve had a few file permission issues under Windows 7, I’m fairly sure it is down to me messing around. However, it is clear that Windows Vista and Windows 7 are both rather sensitive to permissions issues which is worth bearing in mind.

Looking at %USERPROFILE%\AppData\Roaming\ActiveState\KomodoEdit, I could see that SYSTEM, my user and Administrators all had full access but that Administrator was the owner of some of the files. I can only summise that this is the issue.

As this needs rebuilding with Windows 7 RC, I haven’t the time to test further but certainly running as Administrator does the trick.

By the way, Komodo Edit is available for Mac and Linux as well as Windows. It has a big brother “Komodo IDE” with additional features if you need them. Both are highly configurable, support many languages and are based on the Mozilla code (like Firefox and Thunderbird) & can easily be extended with JavaScript.


Technorati : , , ,
Diigo Tag Search : , , ,

How-to show a Message of the Day (MOTD) at the Windows Command Prompt

One of the features available under UNIX is the Message of the Day (MOTD). This is run every time you start a command prompt and displays the content of a file. In addition, the UNIX shells allow all sorts of stuff to be run and configured every time you start a new prompt using the .profile and .bashrc command files.

Windows users don’t generally expect that kind of flexibility from their command prompts. However, Windows does indeed support the use of an “autorun” into which you can shoe-horn any command you like.

So for my standard setup, I make the shell autorun run a “.profile.cmd” file that sits in the %USERPROFILE% folder. From that file, I can run anthing I like.

To set up a shell autorun, you have to edit the registry so the usual warnings to be careful and back things up apply. There are two locations you can set, one for the machine as a whole and one for the logged-in user.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Command Processor

If you want to set an autorun for another user, you need to go down HKEY_USERS and find the appropriate one, it’s really easier just to log in!

In one or both of those locations, add a new “String Value” (REG_SZ) called “AutoRun” with the value:

%USERPROFILE%\.profile.cmd

Now create that file and put in a message such as:

@echo "Hello and welcome to my command prompt"

Save the file and open a new shell and you should see the message just after the Microsoft copyright.

This should work on all versions of Windows at least from XP onwards.

If you want to add this to a batch file to set up new machines, here is the command you need:

reg.exe ADD "HKLM\SOFTWARE\Microsoft\Command Processor" /v AutoRun /t REG_SZ /d ^%USERPROFILE^%\.profile.cmd /f

(Note that the above needs to go on a single line)


Technorati : , , , , ,
Diigo Tag Search : , , , , ,

How-to use WebDAV on Windows Vista and Windows 7

Windows is supposed to have a built-in WebDAV client. However, it doesn’t ever seem to have been especially robust and certainly since Vista a lot of people (myself included) have found that it simply doesn’t work on many supposedly WebDAV enabled sites.

Thankfully there are a couple of free (and some not free) options that, while not as nicely integrated into Windows Explorer, do enable you to transfer files back and forth.

The two that I found were:

I’ve tried the first of these two and it seems to work OK. It’s not polished but it gets the job done. There is also an online version using a Java plugin if you don’t want to install the client.

It seems that the second option is no longer in active development. Don’t be fooled by the web site though, they have changed the license to freeware.

Now, at last, I can once again load multiple files to our corporate intranet on WebEx without having to switch to a Linux desktop!

UPDATE 2012-06-20: Further research shows that Windows 7 (and Vista) can be made to work.

The trick is to use digest authentication instead of basic authentication on the WebDAV server. The problem then being that it may no longer work with pre-Vista clients. It is possible that you would need to set up two virtual links to the same folders, on with basic authentication for pre-Vista clients and one with digest authentication for Vista/Win7 clients.

It is also worth noting that http://support.microsoft.com/kb/907306 is supposed to fix this issue for Vista but shouldn’t be applied to Windows 7.

Anyspeak.org has some additional information on making Windows 7 recognise WebDAV servers that only have basic authentication available. However, this requires admin access on the client PC.

Technorati : , ,
Diigo Tag Search : , ,

Keeping Control: File and Folder Links for Windows Users

A good backup strategy for any computer involves keeping control of where stuff is stored. The fewer locations that contain files that change, the fewer locations have to be maintained.

UNIX users have always had the ability to keep things wherever they wanted and then to LINK that information into the required location. Basically, links create a link or tunnel between one file or folder and another. Most of the time, you will not notice that you’ve entered a tunnel and you are not interested really.

Windows users, however, have always been the poor cousins here. Stuck as we were in FATland, we had no access to fancy features such as links. So Microsoft in their inimitable fashion created a poor-man’s link – the Windows Shell Shortcut – so that the Windows GUI had some minimal capability (really only for menu’s and Windows Explorer).

Windows 2000 improved on this by introducing “Reparse Points” one form of which is the “Junction“. This is an extension to NTFS that allows folders to be joined (linked) to another location in the local volume space. Making junctions is not an obvious process, you can do it from the disk manager and there is a tool in the Windows 2000 server resource kit called linkd. The POSIX tools included in the resource kit contain the UNIX command ln which can also create junction points and hard links; fsutil in XP can also. There are some third party tools too.

It’s odd because I seem to remember that OS/2 had some kind of linking feature.

Anyway, links of the UNIX type are a massively useful feature that has finally (with Vista, Windows 2008 and beyond) made it fully to NTFS and Windows.

Vista, Windows 2008 and Windows 7 all have a command line tool called mklink. This can be used like the Unix ln command to create both hard links (which must be on the same volume) and soft links. Soft links under Windows can, in fact, span across SMB network drives as well.

You might also like to look at another free tool called “Link Shell Extension” by Hermann Schinagl. This integrates into Windows Explorer, the web site also has a more complete explanation of the history of links in Windows. LSE does a number of clever things and is well worth a look. Hermann also has a “dupmerge” tool on his web site that will replace duplicated files with hard links.

So now, if we want to tweak the HOSTS file for example (c:\windows\system32\drivers\etc\hosts), we don’t need to leave in place since that would mean that we would need an extra backup routine. Instead, copy it to somewhere that already gets backed up. Delete the original file and then from the command line:

mklink c:\windows\system32\drivers\etc\hosts %USERPROFILE%\BACKUPS\hosts

Now you can edit the hosts file from either location, there is only one file (in %USERPROFILE%\BACKUPS). The difference being that even if you delete the file from its normal location, it will still exist in the “real” location. If you delete it from its “real” place in BACKUPS of course, the link will be broken and wont work.

To link a complete folder, it is the same command with a /D parameter added. For example, I keep a folder of command line utilities such as ls, ssh and rsync in a folder on a USB pen drive. I sync that folder to the BACKUPS location on my hard drive for convenience but I need the folder in my PATH otherwise its hard to execute the utilities. I don’t want a really long path, it’s bad enough already, so I link the folder to c:\cmd with the following:

mklink /D c:\cmd %USERPROFILE%\BACKUPS\PEN\cmd

Now I add c:\cmd to the path and the utilities seem to be in both places.

I’ve said in other posts that I like to reinstall Windows now and again but it can be a pain to restore all of the document files. Similarly, if you keep multiple operating systems on your hard drive, how do you keep your documents sorted? One way is to put all documents, videos, music, etc. onto a separate partition. Now, instead of going mad with the Windows registry trying to relocate your normal documents folders to another drive. Simply delete the normal documents folder – %USERPROFILE%\Documents\ under Windows 7 and relink it to the appropriate folder on the other drive as so:

mklink /D %USERPROFILE%\Documents d:\Docs

Put this in a script that you run when you reinstall Windows and its easy and quick.

One final note. You may find a few pieces of software that cannot cope with links. Certainly Subversion cannot though Bazaar can. Windows Explorer seems OK though as do utilities such as RSYNC.


Technorati : , , , , , ,
Diigo Tag Search : , , , , , ,

Windows 7 (Build 7000) still my main desktop OS

A quick update on Windows 7. I’m still using the version from MSDN, Build 7000.

Unfortunately, it will not let me report bugs for some reason, the “Send Feedback” link always fails to authenticate my Windows Live login. However, I do have a few issues and a few likes.

Issues with Windows 7

Possibly Windows 7 Beta Issues

  • Power options only shows 2 out of 3 std options
  • Creating new folder in all users (in explorer), creates new folder but doesn’t allow rename (says folder already in use) yet allows delete
  • Screen resolution seems to randomly reset to something lower (NVIDIA WDM driver). Often can’t be put back to native resolution again without either a reboot or lots of tries and messing. Often happens after UAC prompts, sleep and logon screensaver, also sometimes when plugging in an external monitor. Trying to reset often causes Windows to go to a LOWER resolution that it started with (1024×768 which seems to be some kind of default)
  • Some folders that are accessible from Windows Explorer main interface give “Access Denied” when accessed from file open/save dialogues
  • Windows update failed from behind ISA proxy when using a proxy.pac file even though IE was working OK
  • Send Feedback failed from behind ISA proxy (even though IE works OK) – both on proxy.pac and direct proxy setting – gives invalid username/password. Actually, this always fails no matter what, it is not just a proxy issue as far as I can tell
  • Windows Mobile 6.1 device not recognised (HTC Touch HD) (Workaround: Install Windows Mobile Centre for Vista)
  • Shift-Delete on a picked file does not always delete file (shift-RMB/Delete does)
  • Windows Explorer does not always refresh correctly (e.g. Duplicate file & group entry [grouped by date], files not removed after delete), clears after F5/Refresh. Sometimes the refresh is very slow (2-3 seconds)
  • Wmic command line tool does not work correctly. The examples in Technet do not work if you want to specify an output template. The templates are not in %systemroot%\system32\wbem (note that the example given in Technet incorrect has a leading \) as expected by wmic, instead they are in a sub-folder, .\en-US and wmic cannot find them. You cannot put in a path either it seems. Copying the template to the parent folder does work. Interestingly, the “files” in the sub-folder are actually linked to sub-folders of %systemroot%\winsxs. Reported via technet.

Fundamental Windows issues still around in Windows 7

  • Can’t temporarily or quickly show hidden files and folders (for admin) – have to use the sledge-hammer approach of changing the Explorer settings which means drilling down several menus (NB: There is a script someone has written to toggle this setting)
  • When files/folders in use, Windows still doesn’t say who or what is using them! (e.g. USB)
  • Non-resizable dialog boxes (e.g. Power options/Advanced Settings, Driver Installation, …) – Even IE8 suffers from this (e.g. Customize Toolbar)

    This really is unacceptable in the 21st century!
  • Disk Management still does not allow a partition (at least the boot one) to be moved. It can be shrunk and can expand if there is space AFTER it but not if the space is BEFORE it
  • Add/Remove programmes – this is an area that Linux has really triumphed over Windows. Windows still has no standard way of keeping applications up-to-date. Also ARP still doesn’t enforce proper clean-up of data (especially the registry). At least this process is vastly quicker than previously
  • Mouse wheel scroll actions are still inconsistent (though better than previously). Some apps, you have to click in a list before the wheel will scroll (e.g. Most native Windows dialogs – add/remove programmes, Explorer) & still scroll even when the mouse is not over the list, others (e.g. Office) need only have the parent window active and don’t scroll when the mouse is not over the list
  • Still cannot easily manage NTFS links (hard/soft/parse points) even though they have been natively supported since Windows/NT! Hint: Use Link Shell Extension (though see MKLINK command line tool as well)
  • Still cannot natively mount an ISO image file (even though VHD files are now supported). Still cannot create an image natively. At least you can burn an image to CD/DVD natively now.
  • It’s inconceivable that, in this day of multiple monitors, Windows still can’t have different backgrounds on different monitors. At least there are some handy Windows shortcuts now to move windows between monitors

So, if those are the issues, what about the likes?

What I like about Windows 7

  • It is fast! Certainly a lot faster than Vista (not hard). It’s quick to start up and shut down as well
  • Left hand pane of Windows Explorer is now much better organised
  • UAC not quite as intrusive (though see issues)
  • Task bar – wasn’t sure I would like this but it does work well
  • Explorer Libraries – Nice little tweak to Windows Explorer, adds the ability to add many folders to a virtual folder and set one of them as the default write location

Hmm, that seems as though I dislike Windows 7? Well actually that impression is wrong. Windows 7 really is what Vista should have been. Lets forget Vista shall we, it was a mistake rather like Windows ME.

I’m still using Windows 7 even though I do get some problems so that should say something. If Microsoft and other key partners – hello NVIDIA! – can get the final issues sorted out, this is going to be a crackingly good operating system for any PC built in the last five to ten years.


Technorati : ,
Diigo Tag Search : ,

How to get and use your local IP address in a Windows 7 (and Vista) batch command file

If, like me, you spend a lot of time on a variety of customer sites, you will probably be familiar with the issues around swapping networks.

I’ve already blogged about the problems with Windows 7, Vista and Firefox proxy settings and I will do some more articles on getting on with problematic proxies later. However, I wanted to let people know how to get hold of your IP address from within a batch (command) file.

@REM ipconfig | find "IPv4 Address"
@REM Find the IPv4 address from ipconfig (13th var if string is split by both @REM and space)
@for /f "usebackq tokens=13 delims=: " %%i in (`ipconfig ^| find "IPv4 Address" `) do @(
   @set MyIP=%%i
   @REM Split addr into components so networks can easily be checked
   @for /f "usebackq tokens=1-4 delims=." %%a in ('%%i') do @(
      @REM @echo %%a,%%b,%%c,%%d
      @set MyIP1=%%a
      @set MyIP2=%%b
      @set MyIP3=%%c
      @set MyIP4=%%d
   )
)

Note the space after the : on line 3. The FOR command used twice here splits the text output from what is in the () after the “in” using the defined delimiters (“:” and space in the 1st case, “.” in the 2nd). In the first FOR statement, we take element number 13 only, it ends up in variable %%i. In the second case, we take elements number 1 to 4, they go into variables %%a, %%b, %%c, %%d.

Now you have not only the full address but also the componants so if you wanted to check whether you were in a particular class C network, you could do something like:

@set corp_addr_ip4=10.97.100.0

@REM == ADD NEW ADDRESS CHECKS HERE ==
@if "%MyIP1%.%MyIP2%.%MyIP3%.0"=="%corp_addr_ip4%" @goto DOSOMETHING

That would check if your local IP address is between 10.97.100.1 and 10.97.199.254

Note that I’ve used the enhanced FOR statement – the “usebakq” makes the FOR statement more like a UNIX type one where commands are enclosed in back-quotes (`). This certainly works for Windows 7 and should, I think, work for Vista. Prior to Vista, you would need to do things differently anyway. At the very least you would have to search for “IP Address” in the FIND statement as IPCONFIG didn’t include IPv6 information.

Now that you have everything in place, you can control the proxy settings for each application that needs access out of the local network. I’ll blog about that another time.


Technorati : , , ,
Diigo Tag Search : , , ,

Windows 7 supports IPTC in JPEG Picture Files! (Not quite – Adobe XMP actually)

Wow! I’ve just discovered by accident that Windows 7 beta supports a few IPTC XMP attributes in picture files. At last, Microsoft supporting standards!

Above is a screen shot from the properties of a test picture. The Description and Origin sections seem to be standard IPTC fields and I checked them out using iTag.

In iTag the Title attribute comes out as both the Title and the Description. The Subject field doesn’t seem to be recognised nor does the comments field. Rating, Tags, Authors and Copyright are all recognised by iTag.

I’ll do some more extensive testing when I get time.

Update: I spoke too soon 🙁 In fact, it’s rather more complex.

Using ExifTool with ExiftoolGUI, I can see that actually, Windows 7 sets BOTH some EXIF attributes and some XMP (the Adobe meta data format) attributes but NOT IPTC attributes. As it happens, iTag also understands these. Here is a table of what seems to get set.

Windows 7 Attribute EXIF Attributes XMP Attributes iTag Name (Note that iTag synchronises EXIF and XMP attributes on updates [may not be a good thing], Win7 overwrites!!!)
Title XPTitle, UserComment,

ImageDescription
Title,

Description

(Note that only the Title will show the title in Windows 7)
Title,

Description (updates EXIF ImageDescription & XMP Description, multi-lines joined with “…”)
Subject XPSubject N/A N/A
Rating (Number, 1-5) Rating (1-5), RatingPercent (as per XMP)

Rating (1-5), RatingPercent (1=1, 2=13)

(Note that Rating alone DOES show the rating stars in Windows 7)

Stars under the thumbnail
Tags (“:” separated) XPKeywords (“:” separated) LastKeywordXMP (separated by “, “)

(Note that this alone DOES show in Windows 7)
Tag Bucket
Comments (Multi-line – ctrl-enter) XPComments (Lines separated with “…”) N/A N/A
Authors Artist, XPAuthor

Creator

(Note that this alone DOES show in Windows 7)

Author
Copyright Copyright Rights Copyright

The “XP…” EXIF attributes seem to be Windows specific as ExifToolGUI doesn’t offer an edit feature for them, just lists them.

So, a rather typically Microsoft mixed bag. Why wasn’t the XMP Description attribute mapped to Windows Comments? Similarly for EXIF UserComment!

Still, it is something anyway and hopefully the table will help you choose which attributes to use for the best cross-tool support.

I will try to add some more applications to the table if I get a change, if you get there before me, please let me know and I will add the details here.

Update 2: A quick look at Wikipedia shows that Microsoft seem to be backing XMP as support is built in to a number of their photo tools. In the past I’ve stayed away from this as only Adobe and other expensive products supported it whereas IPTC had more widespread support. It seems that this may be changing now – typical – time perhaps to find a way to copy all those IPTC attributes in my photos across to XMP.

See my note about what happens with Windows 7 and iTag updates. This is BAD. Windows 7 rides roughshod over several attributes. Even worse, iTag was unable to reopen the file after Windows 7 had updated it.

So do not use Windows 7 to update image attributes if you also want to use other, more professional tools.


Technorati : , , ,
Del.icio.us : , , ,
Diigo Tag Search : , , ,

Windows 7 Beta – Now my main OS

I’m now using Windows 7 Beta (Build 7000) as my day-to-day operating system.

It is generally very well behaved I have to say and appears to be what Vista should have been from the start. Vista reminds me a lot of Windows/ME, anyone remember that? Another failed Windows build. In reality, Vista was the Windows 7 beta.

Of course, there are a few rough edges and I’ll do a post about them shortly.

The PC is a Dell M1710, 17″ screen and 4GB RAM with WiFi and an NVidia 7900 series mobile graphics card.

proxy.pac files, Mozilla (Firefox & Thunderbird) and Vista or Windows 7

I’ve found a problem with Mozilla based products and proxy settings.

To automatically configure a proxy for use by Internet browsers, you can use a file called “proxy.pac“. This is a JavaScript function that is loaded into the browser when it starts and redirects requests via a proxy where required.

All of the descriptions for this file you will find on the Internet will provide the following example of checking your current IP address. This is used for laptops where the IP address will change depending on where you are. You can check when the laptop is on the corporate network and redirect requests via the corporate proxy as needed:

if ( isInNet(myIpAddress(), "10.10.10.0", "255.255.255.0") ) {
  alert("Corporate address & proxy");
  return "PROXY 10.10.10.240";
 }

Well, this does not work for Mozilla based applications (for example Firefox and Thunderbird) if you are using Microsoft Vista or Windows 7.

That is because, under those operating systems, the internal function myIpAddress() does not return an IPv4 address as expected (e.g. 10.10.10.5) but an IPv6 address instead (e.g something longer with lots of “:”).

In order to make your proxy.pac file work with both IE and Mozilla, use something like:

if (isInNet(myIpAddress(), "10.97.100.0", "255.255.255.0") ||
            shExpMatch(myIpAddress(), "fe80::b892:6a74:9635:*") ) {
  alert("Corporate address & proxy");
  return "PROXY 10.61.9.200:8080; DIRECT;";
 }

You can discover your IPv6 address in several ways but the trusty command line “ipconfig /all” shows you everything you need.

See also this discussion on the Mozilla support forums: proxy.pac myIpAddress() returns incorrect format?


Del.icio.us : , , ,
Technorati : , , ,
Diigo Tag Search : , , ,