Switching Linux Mint on Thinkpad T61p

I've been wanting to try and switch to Linux full-time on my home desktop/laptop machines for a while, and the amount of spyware / tracking / report back in Windows 10 is pushing me to make a real effort this year.  So, I'm taking my old Thinkpad T61p and putting Linux Mint 17.2 on it.  My requirements were:

• Full disk encryption
• Virtualization for a Win7 or Win10 guest
• Running as much as possible inside of Linux
• 64bit OS

A lot of my day-to-day software is already multi-platform or open-source.  For instance:

• Mozilla Firefox and Google Chrome for web browsing
• Mozilla Thunderbird for email
• SVN or git for version control
• GPG for encryption
• Pidgin for instant messaging and IRC
• Audacity, Handbrake, VLC for multimedia work
• About half of my Steam games are Linux / SteamOS
• OpenVPN
• pgAdmin III
• Synergy

The troublesome applications will be:

• Microsoft Office is iffy under WINE
• Microsoft Access does not run under WINE
• Skype might be a problem
• ODBC connections for use with MSAccess
• The half of my Steam games that are not cross-platform
• Cisco AnyConnect VPN client for access to my virtual servers

Because I'm doing this on my old laptop (until the T550p comes out later this year), I'm not too worried about video games that won't run under Steam for Linux.  The only games that this old Core2Duo system will support are the lighterweight / older games that don't need much CPU or video graphics performance.

Linux server partitioning with mdadm software RAID and LVM2

Over the years, I've really come to appreciate what judicious use of LVM (or LVM2) brings to the table when administering servers. If you rely on it heavily and leverage it properly, you can do things like:

• Snapshot any file system (other then /boot) for backups, or to make images, or to test something out.
• Migrate a logical volume (LV) from one physical volume (PV) to another, without having to take the file system offline or deal with downtime.
• Resize file systems that are too large or too small, with minimal downtime (if the file system supports it).

Basically, other then /boot, if you're thinking of creating a separate partition or Software RAID device the you should be using LVM instead of physical partitions or RAID devices. You gain a lot of flexibility in the long-run and setting up LVM on top of hardware or software RAID or plain old disks is no longer that difficult.

These days, when I setup disk partitions to hold a server's boot-up files, I only create (2) partitions on the drive. One for the Software RAID-1 mirror set to hold /boot (usually 256-1024MB) and the rest of the drive is a second RAID-1 mirror set that is turned into a LVM physical volume (PV) and assigned to a volume group. I will usually only partition out to about 99% of the drive size if I'm doing Software RAID, because that makes it easier later to put in a different model disk of the same capacity and still have things work. Drives from different manufacturers have slightly different capacities, so you can run into trouble down the road when you go to replace a failed drive if you assumed all drives were exactly the size as your original drives.

Inside that LVM volume group (VG), I create all of my other partitions. These days, that means:

• / - the "root" partition, usually 16-32GB for CentOS 6
• /home - Usually starts at 4GB for a server where people won't be logging in much.
• /opt - 4-24GB
• /srv - 1-4GB (sub-directories get their own LV later)
• /tmp - 4-24GB
• /usr/local - 8-24GB
• /var - 4-24GB
• /var/log - 4-24GB
• /var/spool - 2-4GB to start
• /var/tmp - 2-8GB to start

And that's just the basic operating system file systems. For things like squid, e-mail, web servers, samba shares, etc., each of those will get its own LV, allocated from the server-wide volume group.

Follow ups:

• GRUB2 understands mdadm (software RAID) and LVM. So we will eventually be able to put /boot in an LVM volume. But the GRUB that ships with RHEL6 and CentOS6 is still GRUB 0.97.

Unsupported sector size 520

While setting up a new DIY server using 15k RPM SAS drives, I ran across some drives which came from the factory with 520 byte sectors (oops).  You'll be able to hook them up and see that they show up as /dev/sdX, but you won't be able to do anything with them (such as read/write or partition).

If you look in /var/log/messages, you will see useful information:

Mar  9 08:08:54 vhc-carthage kernel: sd 6:0:7:0: [sdh] Attached SCSI disk
Mar  9 08:08:57 vhc-carthage kernel: ...ready
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] Unsupported sector size 520.
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] 0 512-byte logical blocks: (0 B/0 B)
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] 520-byte physical blocks
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] Write Protect is off
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] Write cache: disabled, read cache: enabled, supports DPO and FUA
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] Unsupported sector size 520.
Mar  9 08:08:57 vhc-carthage kernel: sd 6:0:8:0: [sdi] Attached SCSI disk

The fix for this is pretty simple in RHEL6/CentOS6.  Major pointers for this came from PissedOffAdmins.

# yum install sg3_utils

# sg_scan -i
/dev/sg8: scsi6 channel=0 id=7 lun=0
    NETAPP    X287_S15K5288A15  NA00 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
/dev/sg9: scsi6 channel=0 id=8 lun=0
    NETAPP    X287_S15K5288A15  NA00 [rmb=0 cmdq=1 pqual=0 pdev=0x0]

Now you should format the offending drive using the "sg_format" command.

[root@vhc-carthage /]# sg_format --format --size=512 /dev/sg8

    NETAPP    X287_S15K5288A15  NA00   peripheral_type: disk [0x0]
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=573653847 [0x22314357]
  Block size=520 [0x208]

A FORMAT will commence in 10 seconds
    ALL data on /dev/sg8 will be DESTROYED
        Press control-C to abort

After this, the disk should work when connected to a regular SAS controller (in my case, a 16-port LSI).

Final note: You will probably find that the disks which have been converted from 520 byte sectors to 512 byte sectors will not perform as well as a disk that was originally manufactured for use as a 512 byte sector drive.  At least, that has been my experience with a sample size of just two drives (and I don't plan on purchasing the wrong drives again).

GNU parted - GTP partition editor

Personally, I dislike "parted" because it works differently then all of the older utilities like fdisk and sfdisk.  Under the old system you could easily "fdisk -l /dev/some/device" and get a sensible reply.  If you try something similar with GNU's parted like "parted -l /dev/some/device", you get a listing of *all* of the devices on the system rather then just what you want to look at.  The syntax is instead "parted /dev/some/device print", and the device name has to come *first* in that command, otherwise it will complain about "print" not being a device.

So, you have a big disk, and you need to partition it, first let's look at the surviving disk in the Software RAID array:

# parted /dev/sde print

Model: ATA Hitachi HDS72202 (scsi)
Disk /dev/sde: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  2000GB  2000GB               primary  raid

Information: Don't forget to update /etc/fstab, if necessary. 

Now, let's look at the new disk:

# parted /dev/sdd print
Error: Unable to open /dev/sdd - unrecognised disk label.

Makes a lot of sense, doesn't it?  Real helpful! Good old GNU parted refuses to give us information about an entirely blank disk, not even helpful information like size (which "fdisk -l" will give us).

So go ahead and label that blank drive with a GPT disk label:

# parted /dev/sdd mklabel gpt

Now we'll actually get good information out of GNU parted's print command:

# parted /dev/sdd print     

Model: ATA ST2000VX002-1AH1 (scsi)
Disk /dev/sdd: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

Well, that's a start (but why couldn't it tell us that before?).

# parted /dev/sdd

(parted) mkpart primary 1 -1

(parted) set 1 raid on

# parted /dev/sdd print

Model: ATA ST2000VX002-1AH1 (scsi)
Disk /dev/sdd: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  2000GB  2000GB               primary  raid


 

Changing "ls" color so that directories don't show up in dark blue

One of the annoyances with Linux if you're using a color terminal (or a color terminal emulator like SecureCRT or PuTTY) is that the "ls" (list directory contents) command has a default of dark blue for directory entries.  Which is fine if you're using a white background, but doesn't work well if you prefer an old-school black background for your terminal.

Fortunately, this is configurable with the LSCOLORS environment variable.  The default string is "exfxcxdxbxegedabagacad" for OS X.  However, on Linux, it looks more like you may be able to configure this using /etc/DIR_COLORS instead.

On my CentOS 5.4 machines, DIR_COLORS lists the entry for directories as "DIR 01;34", which means that it uses a bold blue on the default background.  One option to ensure a bit more contrast would be to change this to always print on a white background.  So you would change this entry to look like:

DIR 01;34;47 # NEW default is Bold blue with White background

References:

change ls colour (color) in bash - GlowingApple  explains back in 2005 how to change colors on the MacOSXHints forum

COLORS Lscolors - another page that explains the LSCOLORS variable

Linux: Checking the fstab prior to a reboot

One of the joys of working on a server from a remote location is dealing with the issue caused by a broken /etc/fstab file. Even the best admins make mistakes and mistakes in that file can lead to a server that won't boot.

Which is fine; if you have an IP-based KVM where you can get console access without actually being at the facility. But not so great when a screwed up fstab file requires you to go physically visit the location.

So how do we verify that our fstab file makes sense prior to a reboot? The answer lies in the mount command. There are (3) useful options that can be passed to the mount command which will help us check the fstab file prior to a reboot.

-f - FAKE IT (Causes everything to be done except for the actual system call). This tells mount to do everything, but don't actually change anything.

-a - Mount all filesystems mentioned in fstab.

-v - Be verbose about it.

Here's an example where everything is fine and dandy.

# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
mount: /dev/md4 already mounted on /var/log
nothing was mounted

Same example, except that I screwed up the name of /dev/md4 in the fstab file:

# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
/dev/md4x on /var/log type ext3 (rw,noatime)
nothing was mounted

Now, there's probably a better way to do this, but this serves as at least a moderate check against shooting yourself in the foot.

Methods ofr remote GUI control of Linux servers

There are currently (3) basic methods for getting a remote control GUI on a Linux server (like we do with Remote Desktop for Windows servers):

1) X-Windows over TCP/IP

All GUI operations on Unix/Linux are handled by the X-Windows sub-system. Window interfaces like KDE, Gnome, and others are merely layered on top of the X sub-system. One of the useful things about X is that any window can be forwarded over TCP/IP to any other X server. So you could run an application on the linux server, but display the output window on your PC (as long as you run a local X server program).

The downside of all this is that accessing remote servers requires the use of SSH port forwarding, and a bit of arcane magic. It's nowhere near as clean of a solution as RDP (Terminal Services). But it can be ultra-secure (by using SSH keys) and it works fairly well across the WAN.

2) VNC

VNC is a screen-scraper solution for GUI desktops, very similar to the old pcAnywhere and e/pop solutions that we used to use.

The downsides of VNC are:
- security is non-existent in the base spec
- different VNC server use different encryptions
- authentication tends to be done via plain text passwords
- rather slow across the WAN

3) NX/FreeNX

A company called NoMachines came out with a different solution called "NX". NX is a protocol that is very similar to RDP and the client works rather similar to Remote Desktop. You used to have to pay for the product, but over the years, they've opened up the source code. So now there are (3) different server implementations (NX, FreeNX, and another) and you can download the NX client from NoMachines for free.

The big advantage here is that security is better and performance is better over slow WAN links.

My new preferred disk layout for servers

With age comes wisdom? After working with SoftwareRAID and Linux servers for a while, I've changed my preferred disk system design and layout.

RAID

Under the old system, I was running a (2) disk RAID1 (mirror) with a hot-spare disk setup and ready for action. But if you're going to have a hot-spare dedicated to the RAID1 array, why not use it as an active array member? That way, if a disk fails, you still have two good disks. Unfortunately, when a RAID element fails, the load from the rebuild process can often kill the one of the remaining disks in the array.

Is it a likely scenario? Probably not. But Linux's Software RAID handles a triple-active RAID1 mirror without any slowdown, so there's not much reason *not* to implement it that way. Plus it's a useful trick to know for situations where you really *do* need to be that paranoid.

(I'm not sure whether any hardware RAID cards provide for a triple-active mirroring RAID1 configuration.)

Partitions

I've also simplified how many partitions I like to have on the disk. My current disk layouts typically look like:

/dev/sdX1 - /dev/md0 - 250MB - /boot
/dev/sdX2 - /dev/md1 - 12GB - / (primary root)
/dev/sdX3 - /dev/md2 - 12GB - / (backup root)
/dev/sdX5 - /dev/md3 - 32GB - /var/log
/dev/sdX6 - /dev/md4 - 2GB - swap
/dev/sdX7 - /dev/md5 - 64GB - /backup/system
/dev/sdX8 - /dev/md6 - (remainder) - LVM area

During normal operations, we boot and run /dev/md1 as our / (root) partition. The /dev/md2 partition is kept offline and is never mounted. Periodically, after validating that the server is in good health, we will copy the contents of /dev/md1 to /dev/md2, make adjustments to /etc/fstab and the hostname. This requires some server downtime (long enough to setup the 2nd root partition).

In the case where the primary OS is hosed, we can boot from the backup OS partition and get back up and running quickly. That gives us the luxury to continue operations until we can schedule downtime to fix the primary OS partition.

Notice that I've broken /var/log out to its own partition. I do this so that an overflowing set of logs won't take the server box down. Plus, by putting the log files in their own physical partition, it's easy to use a boot CD or USB key to gain access to the logs in case of severe issues.

The other physical partition that I consider necessary is /backup/system. This partition is used to hold images of the boot and root partitions, along with information about the partition layout and images of the MBRs. Basically, it's used to store disaster recovery backups. You should not have this partition mounted during normal operations. Taking the contents of this partition offsite is also a good idea. A basic text file of how the backups were created along with information for how to restore these backups is recommended.

Summary

This setup tries to walk the fine line between keeping it simple, but having enough flexibility to deal with a large set of potential failures. Anything from a two-disk failure, to the primary OS being hosed, to both OS partitions having problems all the way up to boot records or the /boot partition being killed.

lm_sensors and Gigabyte GA-6VA7+ (take 2)

A while back I had tried to configure lm_sensors on an old Gigabyte GA-6VA7+ motherboard. That didn't work out so well, so I'm going to give it another shot. Which is important because I had a drive overheat this week due to a failed fan.

There's a new version of lm_sensors out (2.10) while I'm still running the old 2.09. Most of the steps are the same, I'm simply emerging the new version and then following the instructions. On my slow little 566Mhz Celeron box, this takes a while. Especially since I'm also rebuilding a failed raid element (thank goodness for Software RAID).

Output of the end of the emerge process:

>>> /etc/init.d/fancontrol
*
* Next you need to run:
* /usr/sbin/sensors-detect
* to detect the I2C hardware of your system and create the file:
* /etc/conf.d/lm_sensors
*
* You will also need to run the above command if you're upgrading from
* <=lm_sensors-2.9.0, as the needed entries in /etc/conf.d/lm_sensors has
* changed.
*
* Be warned, the probing of hardware in your system performed by
* sensors-detect could freeze your system. Also make sure you read
* the documentation before running lm_sensors on IBM ThinkPads.
*
* Please see the lm_sensors documentation and website for more information.
*
>>> Regenerating /etc/ld.so.cache...
>>> sys-apps/lm_sensors-2.10.0 merged.

sys-apps/lm_sensors
selected: 2.9.2
protected: 2.10.0
omitted: none

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging sys-apps/lm_sensors-2.9.2...
No package files given... Grabbing a set.
--- !mtime obj /usr/share/man/man8/sensors-detect.8.gz
...
(snip)
...
--- !targe sym /usr/lib/libsensors.so
>>> Regenerating /etc/ld.so.cache...
>>> Regenerating /etc/ld.so.cache...
>>> Auto-cleaning packages ...

>>> No outdated packages were found on your system.


* GNU info directory index is up-to-date.
* IMPORTANT: 14 config files in /etc need updating.
* Type emerge --help config to learn how to update config files.

#

Output of the sensors-detect phase:

# /usr/sbin/sensors-detect
# sensors-detect revision 1.413 (2006/01/19 20:28:00)

This program will help you determine which I2C/SMBus modules you need to
load to use lm_sensors most effectively. You need to have i2c and
lm_sensors installed before running this program.
Also, you need to be `root', or at least have access to the /dev/i2c-*
files, for most things.
If you have patched your kernel and have some drivers built in, you can
safely answer NO if asked to load some modules. In this case, things may
seem a bit confusing, but they will still work.

It is generally safe and recommended to accept the default answers to all
questions, unless you know what you're doing.

We can start with probing for (PCI) I2C or SMBus adapters.
You do not need any special privileges for this.
Do you want to probe now? (YES/no): Yes
Probing for PCI bus adapters...
Use driver `i2c-matroxfb' for device 01:00.0: MGA G200 AGP
Use driver `i2c-viapro' for device 00:07.3: VIA Technologies VT82C596 Apollo ACPI
Probe succesfully concluded.

We will now try to load each adapter module in turn.
Load `i2c-matroxfb' (say NO if built into your kernel)? (YES/no): yes
FATAL: Module i2c_matroxfb not found.
Loading failed... skipping.
Module `i2c-viapro' already loaded.
If you have undetectable or unsupported adapters, you can have them
scanned by manually loading the modules before running this script.

To continue, we need module `i2c-dev' to be loaded.
If it is built-in into your kernel, you can safely skip this.
i2c-dev is not loaded. Do you want to load it now? (YES/no): yes
Module loaded succesfully.

We are now going to do the adapter probings. Some adapters may hang halfway
through; we can't really help that. Also, some chips will be double detected;
we choose the one with the highest confidence value in that case.
If you found that the adapter hung after probing a certain address, you can
specify that address to remain unprobed. That often
includes address 0x69 (clock chip).

Next adapter: SMBus Via Pro adapter at 5000
Do you want to scan it? (YES/no/selectively): yes
Client at address 0x50 can not be probed - unload all client drivers first!
Client at address 0x51 can not be probed - unload all client drivers first!
Client at address 0x52 can not be probed - unload all client drivers first!
Client found at address 0x69

Next adapter: ISA main adapter
Do you want to scan it? (YES/no/selectively): yes

Some chips are also accessible through the ISA bus. ISA probes are
typically a bit more dangerous, as we have to write to I/O ports to do
this. This is usually safe though.

Do you want to scan the ISA bus? (YES/no): yes
Probing for `National Semiconductor LM78'
Trying address 0x0290... Failed!
Probing for `National Semiconductor LM78-J'
Trying address 0x0290... Failed!
Probing for `National Semiconductor LM79'
Trying address 0x0290... Failed!
Probing for `Winbond W83781D'
Trying address 0x0290... Failed!
Probing for `Winbond W83782D'
Trying address 0x0290... Failed!
Probing for `Winbond W83627HF'
Trying address 0x0290... Failed!
Probing for `Winbond W83627EHF'
Trying address 0x0290... Failed!
Probing for `Silicon Integrated Systems SIS5595'
Trying general detect... Failed!
Probing for `VIA Technologies VT82C686 Integrated Sensors'
Trying general detect... Failed!
Probing for `VIA Technologies VT8231 Integrated Sensors'
Trying general detect... Failed!
Probing for `ITE IT8712F'
Trying address 0x0290... Failed!
Probing for `ITE IT8705F / SiS 950'
Trying address 0x0290... Failed!
Probing for `IPMI BMC KCS'
Trying address 0x0ca0... Failed!
Probing for `IPMI BMC SMIC'
Trying address 0x0ca8... Failed!

Some Super I/O chips may also contain sensors. Super I/O probes are
typically a bit more dangerous, as we have to write to I/O ports to do
this. This is usually safe though.

Do you want to scan for Super I/O sensors? (YES/no): yes
Probing for `ITE 8702F Super IO Sensors'
Failed! (skipping family)
Probing for `Nat. Semi. PC87351 Super IO Fan Sensors'
Failed! (skipping family)
Probing for `SMSC 47B27x Super IO Fan Sensors'
Failed! (skipping family)
Probing for `VT1211 Super IO Sensors'
Failed! (skipping family)
Probing for `Winbond W83627EHF/EHG Super IO Sensors'
Failed! (skipping family)

Do you want to scan for secondary Super I/O sensors? (YES/no): yes
Probing for `ITE 8702F Super IO Sensors'
Failed! (skipping family)
Probing for `Nat. Semi. PC87351 Super IO Fan Sensors'
Failed! (skipping family)
Probing for `SMSC 47B27x Super IO Fan Sensors'
Failed! (skipping family)
Probing for `VT1211 Super IO Sensors'
Failed! (skipping family)
Probing for `Winbond W83627EHF/EHG Super IO Sensors'
Failed! (skipping family)

Sorry, no chips were detected.
Either your sensors are not supported, or they are
connected to an I2C bus adapter that we do not support.
See doc/FAQ, doc/lm_sensors-FAQ.html, or
http://www2.lm-sensors.nu/~lm78/cvs/lm_sensors2/doc/lm_sensors-FAQ.html
(FAQ #4.24.3) for further information.
If you find out what chips are on your board, see
http://secure.netroedge.com/~lm78/newdrivers.html for driver status.
#

Mmm, doesn't look good, does it?

The chips that I know are on this motherboard (VIA Apollo chipset series) are:

Winbond 83977 (I/O chipset)
VIA VT82C596B
VIA VT82C693A

Hmm... those VIA chips are supposedly supported via the i2c-viapro module. I'll need to reboot at some point and check out my BIOS settings to make sure the proper things are turned on.

More handy Linux console tools

A few more tools that I've found useful for console-only Linux server administration. All of these have Gentoo ebuilds so they can simply be emerge'd into your system.

atop - Similar to the built-in "top" command in Linux/Unix, except that it gives additional details about what is going on with the system. Of particular use is that it will show you individual disk utilization statistics so that you can locate spindles that are in heavy use. (Which is much needed when tuning a database server.)

tree - Allows you to output a hierachial display of all files and directories on your system in a "tree" view. Which is useful for documenting the directory layout of your system.

Unix/Lunix Filesystem Hierarchy Standard

The Filesystem Hierarchy Standard is a set of guidelines designed to get various distrobutions of Linux/Unix all on the same page with regards to what goes where in the file system.

For us laymen, it's a decent guide to what is where in the filesystem (why is there a /sbin and a /usr/local/sbin?). It helps answer questions like, "If I have a backup script, where should I put the file?".

Linux on Laptops

Today's useful link is: Linux on Laptops.

They have a short-n-sweet index where people submit links to their HOWTO pages regarding how to get a specific distro to work on a particular make/model of laptop.

Linux Bare-Metal Backup and Recovery Planning

The whole concept behind bare-metal restore is that it's the shortest point from pulling a replacement set of server hardware out of a box to having a working system to replace the one that's down. Ideally, without having to go through re-installing the operating system (which, in the case of Gentoo, can take a few hours). It also (generally) only works when you have exactly the same hardware in the replacement box as the box that's kaput. So if you're using any exotic hardware (e.g. a fancy RAID card), you'd best buy (3). That way when the first one dies, you can put in one of the (2) spare cards and still get at your data.

Some backup software makes the process as simple as dropping a boot CD in the drive and inserting the most recent bare-metal backup (sometimes called a "gold" tape?) tape. It can also be a multi-step process, where the first step restores the operating system to a known state and the second step handles restoring the data and applications.

Since I'm not ready to tackle writing a guide, I'll settle for a few links:

Linux Complete Backup and Recovery HOWTO (by Charles Curley) - A very good starting point.

Linux Journal: Bare Metal Recovery, Revisited (by Charles Curley, Aug 2002) - A good sidebar discussion of the HOWTO document that he wrote.

About.com: Linux backup - A collection of links to backup software.

Unix SysAdm Resources: Backup & Archival Software for Unix (Stokely Consulting) - Listing of backup software for unix.

Linux Backups Mini-FAQ (Karsten M. Self) - Good article on simply using "tar".