17 Jun 2005
Backups, rsync and rsnapshot
I've just taken delivery of a spanking new 500GB LaCie Firewire drive for work. I've partitioned it in two, with the bulk of the space allocated for video data, but I've also made a backup partition for my laptop. I've wanted to try using rsync for ages, but I never really had a large enough hard drive spare to try it.
The great thing about rsync is that you can do frequent incremental backups (keeping snapshots of the state of your drive at various time intervals), while not using much more space than a single backup, and while allowing you to easily restore a whole directory structure without having to piece together backup sets.
It manages this feat by using hard links, and rotating backup sets. In the latest set, only the files which have changed are backed up, but hard links are created for the rest of the files, so the latest backup appears to be (and to all intents and purposes is) a full copy of the source.
rsync is included in Tiger, but it can be a little tricky to set up a script to back everything up, particularly if you want to back up multiple but separate directories to a single destination. However, rsnapshot provides a nice, easily configured wrapper around rsync. You can configure the number of hourly, daily, weekly and monthly backups that get rotated, and you can set multiple source paths to backup.
It works very nicely, but I've been trying to think of a good way to automate starting it. Now that cron is deprecated in Tiger, I'm looking in to using launchd to run the backups. The tricky part is that I only want it to run if my Firewire drive is mounted, which is only more or less in office hours. rsnapshot does come with a little script that checks whether the snapshot path is mounted before it runs the process itself, but it would be nice to do it within launchd. There's the watchpaths property which is supposed to only start the process if a path changes (which would happen if my drive was mounted). The other snag is that I don't think that launchd yet supports the '*/4' style of time intervals (i.e. every 4 hours).
I'll try to construct something over the weekend, and test it on Monday. If anyone has any tips for using launchd as a cron replacement, let me know!
2
Congratulations to your new harddrive. 500 GB *shaking my head* My first computer had 40 GB harddrive and 1 MB RAM... I did some "research" when I bought my external harddrive and found that TRI-backup is the program I was looking for. It's very easy to create scripts and you can do nice things like, starting a backup when a volume is mounted. Check it out at: http://www.tri-edre.com/english/tribackup.html3
In rsnapshot you can turn on the "no_create_root" parameter so that rsnapshot won't attempt a backup if the snapshot_root doesn't already exist. If your snapshot_root was at, say, /Volumes/LaCie/snapshots/, that directory wouldn't exist while the drive was unplugged. I don't know anything about launchd, but you could also specify the */4 thing in cron as 0,4,8,12,16,20 so perhaps a similar strategy could be used with launchd. Feel free to drop by the rsnapshot-discuss mailing list if you have any questions or comments!4
500Gb eh - even more room for lazy programming. The whole of the "Automatic Landing Program" code for the HS146 passenger jet fitted into 2k, yes 2000Bytes, about the size of an empty folder under modern operating sysems. If modern programmers wrote in old fashioned machine code you could run the country on 500Gb with room left over for Pacman!by Jonathan Briggs @ 18/06/2005 10:06 am • Permalink
5
my first computer had 32k ram and a 300baud tape drive
my brain is still recoiling from anything remotely technical, but you may glean some nice tips from this discussion re rsync. the JWZ character is renowned unix hackerguru jamie zawinskie who wrote much of netscape and mozilla singlehanded. he finally got jack of linux and last week(i think) bought a mac, and is asking for tips
keychain/rsync tips
(the last straw)
6
Nicholas, I'll take a look. ThoughtBadger: rsync indirectly through rsnapshot. That link seems to bring up a blank page, which is weird. Phil: I do actually have a GUI backup application (Synchronise Pro X), but I was attracted by the rotating snapshots aspects of rsync and rsnapshot. Nathan: Great utility by the way! That's helpful to know about no_create_root. I've added it to my config. I don't think that launchd supports either the */4 form or the comma separated list yet, so as cron is still functional in Tiger, I'm using that for now. Jonathon Briggs: I had the same conversation with someone today about the size difference between Unix binaries and Mac OS X applications. Still, as I said, this drive isn't for programming (lazy or otherwise
), but video files, which *are* massive. I can fairly easily collect a couple of GB of video data per day.
Saltation: Thanks for the links. He has some pretty snarky commenters, doesn't he. 
Glad everyone is more polite here...
7
Sorry bsag my mistake, an extra backslash crept in. Here is the correct link to the rsyncbackup scripts.by ThoughtBadger @ 22/06/2005 9:06 am • Permalink
8
hai i want knowledge of rsnapshot about backup thanks vijayby vijay @ 11/07/2005 1:08 pm • Permalink
9
I'm looking into improving my backup regime as well. One thing you should know about Apple's rsync is that if you use the -E flag to copy extended attributes, then -t (which is implied by -a) won't set timestamps properly. This is a known issue and if you google about, at least one guy has a patch for it. Hopefully Apple will provide a real fix before long. In a lot of ways, I like unison better than rsync, but the Mac version doesn't handle extended attributes yet. Doh. As for the auto-launch, I suspect that there is a way to pull this off using launchd's WatchPaths feature, but all I have found so far is folks who can't quite make it work.10
"my first computer had 32k ram and a 300baud tape drive" pah! thats double what i had. my first computer had 16k of ram - which was huge compared to what my cousin had - he had one of the first ZX81's , with 1k of RAM - yes, you read that right - 1k. or 1024 bytes. http://www.nvg.ntnu.no/sinclair/computers/zx81/zx81.htm -----11
from man launchd.plist StartInterval <integer> This optional key causes the job to be started every N seconds. If the system is asleep, the job will be started the next time the computer wakes up. If multiple intervals transpire before the computer is woken, those events will be coalesced into one event upon wake from sleep.by Kurt Werle @ 20/02/2007 7:21 pm • Permalink
12
(This post is sncredibly old, I know) @Nathan > In rsnapshot you can turn on the "nocreateroot" parameter so that rsnapshot won't attempt a backup if the snapshotroot doesn't already exist But that is actually a problem. If you use rsnapshot on a laptop, that means often snapshotroot will not exist. Launchd will fire up, for example, "rsnapshot monthly", and think it did its job. If the volume was not there, it will not trigger again until one month after. That's why it'd be nice that it was actually launchd who checked whether the volume is there or not, and retry later on whenever the volume is connected. I have no clue of how to do that, though.13
It has been so long that there are now better controls: man launchd.plist ... PathState <dictionary of booleans> Each key in this dictionary is a file-system path. If the value of the key is true, then the job will be kept alive as long as the path exists. If false, the job will be kept alive in the inverse condition. The intent of this feature is that two or more jobs may create semaphores in the file-system namespace. ... WatchPaths <array of strings> This optional key causes the job to be started if any one of the listed paths are modified. QueueDirectories <array of strings> Much like the WatchPaths option, this key will watch the paths for modifications. The difference being that the job will only be started if the path is a directory and the directory is not empty. ... StartOnMount <boolean> This optional key causes the job to be started every time a filesystem is mounted. Clever writing of the launch script and these various settings should get you a very reasonable system. I'm not currently using launchd, so I don't plan on spending much time pondering this.by Kurt Werle @ 22/02/2008 8:46 am • Permalink
1
Also have a look at rdiff-backup. ----- Are you using rsync or rsyncx? Does it make any practical difference if you don't use any OS 9 apps? There is another set of scripts here that might interest you.by ThoughtBadger @ 17/06/2005 11:06 pm • Permalink