Installing borgbackup under Ubuntu Gnome

My favorite file-level backup tool for Linux (or OS X or Cygwin) is still borg (a.k.a. borgbackup).  The features that I rely on are:

• Efficiency when dealing with millions of files, borg is very fast at scanning the file system and figuring out what needs to be backed up.  In the past, I've run it against an IMAP mail server file system with a few million files and about 100GB.  Each snapshot would only take 15-20 minutes instead of a few hours for some other solutions.
• Very few files created on the target file system.  Other solutions like rsnapshot or rdiff-backup will turn 1 million source files into 10 million backup files (or worse) due to how they implement snapshots.  While it's useful to be able to browse the backup directory just like the source file system, it causes all sorts of issues for disk performance or copying backup directories off to removable media.  In contrast, borg creates only a few dozen or few hundred files per snapshot.
• Deduplication using variable block sizes.  This is a huge win if you have a lot of files where parts of the files are identical.  The algorithm in borgbackup will find those identical sections and only store them once in the backup repository.
• Efficiency over the network.  For the most part, as long as borgbackup is isntalled on both the source and destination systems, borg is very good at sending the least amount of traffic over the wire. With variable block deduplication, it's going to be more efficient then other file level deduplication solutions.
• Built-in client-side encryption.  While I don't use this (my backups are stored on LUKS encrypted file systems), this could be useful if you are backing up to a destination server that you do not trust 100%.
• Compression of backup data prior to transmission to the repository server.  This also helps reduce the size of the repository on the target server.
• Works over SSH (as does rsync, rsnapshot, rdiff-backup).

FSVS on Ubuntu 10.04 LTS (Lucid Lynx)

(See also my older post on installing FSVS on CentOS 5.5.)

While I prefer CentOS / RHEL for our servers, I do have a few Ubuntu machines laying around that I use as desktops.  And given my desire to track things using FSVS as much as possible, that means I need to install FSVS under Ubuntu as well.

Note: While you can install fsvs via apt-get with "apt-get install fsvs", the version included right now in the Ubuntu repositories is only FSVS 1.1.17.  This is fairly old code from around 2008.  The latest version is 1.2.3  and was released in January 2011.

Step 1: Create the server user and repository

On our SVN server, we'll need to setup a user account and create a repository to hold the files.  All of our repositories are kept under /var/svn and we create users and groups named "svn-sys-somesystem".  The individual system repository gets named sys-somesystem.

# cd /var/svn

# svnadmin create sys-somesystem

# chmod -R 750 sys-somesystem

# chmod -R g+s sys-somesystem/db

# useradd -m svn-sys-somesystem

# chown -R svn-sys-somesystem:svn-sys-somesystem sys-somesystem

# passwd  svn-sys-somesystem

(give it a very long, very random password)

Changing password for user svn-sys-somesystem.

New UNIX password:

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

# su svn-sys-somesystem

$ mkdir ~/.ssh

$ chmod 700 ~/.ssh

$ cd ~/.ssh

At which point we're ready to paste the SSH key from the other system in.  Switch to the system that you will be adding FSVS to.

Step 2: Setting up SSH keys

Login to the system which you will be adding as a FSVS client.  Under Ubuntu, this means a lot of 'sudo' work.  Note that lines ending in '\' should be concatenated together to form a single command.  You'll need to create a .ssh/config file so that SSH knows how to talk to the SVN server.

$ sudo mkdir /root/.ssh

$ sudo chmod 700 /root/.ssh

$ sudo /usr/bin/ssh-keygen -N '' \

-C 'svn key for root@hostname' \

-t rsa -b 2048 -f /root/.ssh/fsvs-key

Generating public/private rsa key pair.

Your identification has been saved in /root/.ssh/fsvs-key.

Your public key has been saved in /root/.ssh/fsvs-key.pub.

The key fingerprint is:

ff:ee:dd:cc:bb:aa:99:88:77:66:55:44:33:22:11:00 svn key for root@hostname

$ sudo vim /root/.ssh/config

Host svn.yoursvnserver.com

Port 22

User svn-sys-somesystem

IdentityFile /root/.ssh/fsvs-key

$ sudo chmod 600 /root/.ssh/config

$ sudo chmod 600 /root/.ssh/fsvs-key

$ sudo chmod 600 /root/.ssh/fsvs-key.pub

$ sudo cat /root/.ssh/fsvs-key.pub

Copy this key into the clipboard or send it to the SVN server or the SVN server administrator. Back on the SVN server, you'll need to finish configuration of the user that will add files to the SVN repository.

# su svn-sys-somesystem

$ cd ~/.ssh

$ cat >> ~/.ssh/authorized_keys

The line for the SSH key should start with the following, which locks down the SSH key a bit and should only allow it to be used to run /usr/bin/svnserve.

command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty,no-port-forwarding,no-X11-forwarding

So a full SSH key line in the authorized_keys files will end up looking like:

command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty,no-port-forwarding,no-X11-forwarding ssh-rsa (long SSH key) (ssh key comment)

Hit Ctrl-C when finished pasting in the key.

$ chmod 600 ~/.ssh/authorized_keys

Now we can go back to the client machine where FSVS will be installed and test that our SSH connection works.

$ sudo svn.yoursvnserver.com

The authenticity of host '[svn.yoursvnserver.com]:22 ([192.168.0.1]:22)' can't be established.

RSA key fingerprint is 99:88:77:66:55:44:66:33:22:11:00:55:ff:ee:dd:aa.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '[svn.yoursvnserver.com]:22,[192.168.0.1]:22' (RSA) to the list of known hosts.

PTY allocation request failed on channel 0

( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) ) Connection to svn.yoursvnserver.com closed.

If you don't get the SVN pipeline information, then the SSH keys are not configured properly, or you forgot to chmod a file back to 600 (usually the authorized_keys file).

Step 3: Installing FSVS

The FSVS install tarball is available at fsvs.tigris.org.

$ cd /usr/local/src

$ sudo wget http://download.fsvs-software.org/fsvs-1.2.3.tar.bz2

$ sudo tar xjf fsvs-1.2.3.tar.bz2

$ sudo chown -R username:username fsvs-1.2.3/

$ cd fsvs-1.2.3/

Now we are ready to configure and compile FSVS.  The following command will check the environment and tell us whether libraries are missing.

$ ./configure

Since we already know that we'll need to install a bunch of things, here is the apt-get command.  Note that if you need to find a development version of a particular package, then "apt-cache search apr | grep 'dev'" may be useful.

$ sudo apt-get update

$ sudo apt-get install build-essential

$ sudo apt-get install libpcre3-dev

$ sudo apt-get install libaprutil1-dev

$ sudo apt-get install libsvn-dev

$ sudo apt-get install libgdbm-dev

Once all that is installed, the "./configure" should run cleanly.  If it doesn't, then you're probably missing some library and will have to add it.

$ ./configure

$ make

Which will compile and link the FSVS program.

$ sudo cp src/fsvs /usr/local/sbin/

$ sudo chown root:root /usr/local/sbin/fsvs

$ sudo chmod 700 /usr/local/sbin/fsvs

Step 4: Association with the SVN repository

$ cd /

$ sudo mkdir /var/spool/fsvs

$ sudo mkdir /etc/fsvs/

$ cd /

$ sudo fsvs urls svn+ssh://svn.yoursvnserver.com/sys-somesystem/

Step 5: Telling FSVS what to ignore

When constructing ignore patterns, generally work on adding a few directories at a time to the SVN repository. Everyone has different directories that they won't want to version, so you'll need to tailor the following to match your configuration. However, I generally recommend starting with the following (this is the output from "fsvs ignore dump", which you can pipe into a file, edit, then pipe back into "fsvs ignore load"):

group:ignore,./backup/
group:ignore,./bin/
group:ignore,./cdrom/
group:ignore,./dev/
group:ignore,./etc/fsvs/
group:ignore,./etc/gconf/
group:ignore,./etc/gdm/
group:ignore,./etc/shadow*
group:ignore,./etc/ssh/ssh_host_key
group:ignore,./etc/ssh/ssh_host_dsa_key
group:ignore,./etc/ssh/ssh_host_rsa_key
group:ignore,./home/
group:ignore,./lib/
group:ignore,./lib32/
group:ignore,./lib64/
group:ignore,./lost+found
group:ignore,./media/
group:ignore,./mnt/
group:ignore,./proc/
group:ignore,./root/
group:ignore,./sbin/
group:ignore,./selinux/
group:ignore,./srv/
group:ignore,./sys/
group:ignore,./tmp/
group:ignore,./usr/bin/
group:ignore,./usr/games/
group:ignore,./usr/include/
group:ignore,./usr/lib/
group:ignore,./usr/lib32/
group:ignore,./usr/lib64/
group:ignore,./usr/local/games/
group:ignore,./usr/sbin/
group:ignore,./usr/share/
group:ignore,./usr/src/
group:ignore,./var/backups/
group:ignore,./var/cache/
group:ignore,./var/games/
group:ignore,./var/lib/
group:ignore,./var/lock/
group:ignore,./var/log/
group:ignore,./var/mail/
group:ignore,./var/opt/
group:ignore,./var/run/
group:ignore,./var/spool/
group:ignore,./var/tmp/

$ vim ~/fsvs-ignores-201105

$ sudo fsvs ignore load < ~/fsvs-ignores-201105

You can check what FSVS is going to version by using the "sudo fsvs status pathname" command (such as "fsvs status /etc"). Once you are happy with the selection in a particular path, you can do the following command:

$ sudo fsvs ci -m "base check-in" /etc

Repeat this for the various top level trees until you have checked everything in. Then you should do one last check-in at the root level that catches anything you might have missed.