INACCESSIBLE_BOOT_DEVICE

Came home from my weekend trip to find that one of my Windows 2000 servers had crashed with the INACCESSIBLE_BOOT_DEVICE 0x0000007B error message. Apparently, the power went out sometime this weekend because the 2nd server had turned itself off, and when the primary server booted back up it was BSOD'd with the 7B error.

The boot disks are mounted on a Promise FastTrak100 TX2 RAID card (which showed both disks as working). So I wasn't too worried (I also have fairly fresh backups of everything on the RAID array.

How to troubleshoot "Stop 0x0000007B" error messages in Windows 2000 wasn't all that useful.

Troubleshooting Stop 0x0000007B or "0x4,0,0,0" Error was a bit more useful.

The first thing tried was to remove any CDs/DVDs from the drives and reboot (no luck). Then I tried rebooting into safe mode with the command prompt (no luck). Then I tried rebooting to "last known good configuration" (again, no luck).

Next, I booted up the Win2000 install CDs, loaded the RAID driver from floppy and went into (R)epair, (C)onsole mode (puts you at a command prompt in C:\WINNT). A directory listing looked clean, so I did a "CHKDSK" from the C:\WINNT folder. That found some errors, so I redid the command with the /R option ("CHKDSK /R").

Once it had finished checking the boot drive (C:), I rebooted the box and it came up fine. It ran CHKDSK on all of the other drives during boot (finding some minor errors in the 2nd partition that is part of the Promise RAID).

My guess at this point is that when the UPS ran out of juice and the box crashed, it caused some issues with the NTFS file system. One of these days I'll put each of my personal servers on their own UPSs and hookup the "shutdown on low battery" cable.

Win2003 Scheduled Tasks 0x80070005 Error

Trying to setup a backup job (kicked off by a .cmd file) on Windows 2003. I have a special, limited rights, user account created (rather then running the backup job under the administrator account). Everything seems fine until I go to test my scheduled task.

Looking at the log (Scheduled Tasks - Advanced - View Log) will show:

Unable to start task.
The specific error is:
0x80070005: Access is denied.

"Access is denied" error message when you run a batch job on a Windows Server 2003-based computer

As expected, this is a permissions error. Specifically, you need to grant permissions for batch processes to use "cmd.exe".

Installing ssh and rsync on a Windows machine: minimalist approach

Installing ssh and rsync on a Windows machine: minimalist approach

Found this on Slashdot recently... haven't had time to go read it.

Tyan Tiger K8W (S2875)

2CPU.com Tyan Tiger K8W (S2875) - Thread with users of the Tyan Tiger K8W.

I have one of these boards with a pair of Opteron 246s. Great for video work, but I never managed to get Aquamark3 running on it to determine performance.

FLAC audio

Re-ripping my CDs to a lossless format for permanent archival (I had ripped them all as 128kbps a few years ago, then reripped at 160kbps, now I'm going lossless). Here's two links that I found useful.

Comparison of lossless audio codecs - Compares most of the popular lossless codecs such as FLAC

FLAC home page - Free Lossless Audio Codec home page over at SourceForge. Be sure to download the FLAC codec installer for Windows if you want to use FLAC in players like WinAmp.

Right now, I'm probably going to rip at compression level = 5 using Easy CD-DA Extractor's Audio CD Ripper. My M.I.B. CD weighed in at 436MB with FLAC(5) compared to about 485MB with compression level zero. That means I'll be able to archive about (8) albums on a DVD-R (compared to 20-30 using 160kbps MP3). If I had ripped it to 320kbps CBR MP3, it would've ended up as 151MB.

AliveAudio - Getting to know FLAC - Lists information about players and ripping utilities.

OpenSSH for Windows

I've pretty much given up on trying to extract the key bits from Cygwin in order to setup a SSHD server. The OpenSSH for Windows project at SourceForge seems to have what I'm looking for, they just don't have the RSync application included.

For an excellent introduction to SSH, check out OpenSSH for the impatient.

For setting up OpenSSH on a server, go ahead and grab the packages from the OpenSSH for Windows SourceForge project. The version that I'm using at the moment is "setupssh381-20040709". Inside that file you'll find a "setupssh.exe" which will install the packages as well as creating the Windows Service. I like to install my copy to "c:\bin\openssh".

Now open up the "c:\bin\openssh\docs\readme.txt" (or quickstart.txt) and follow the directions in order to create the "group" and "passwd" files. Then start up the OpenSSHD service (either from the command line as shown in quickstart.txt or using the Services control panel).

You should now be setup so that you can SSH in to the server from another workstation and get a command prompt on the server. However, the default install is pretty good in security, so you should not need to change anything sshd_config file. However, some things you may wish to change are:

1) The default server key-length is 1024 bits (which is okay, but not outstanding anymore). The man page says key lengths over 1024 bits don't matter, but another books says you should use 2048 bit keys.

2) Some key variables in the sshd_config file are:

a) PermitRootLogin - should be set to "no" which prevents you from logging in as root from another machine.

b) RSAAuthentication - setting this to no will disable the ability to login with a SSH1 client (I think...). The default sshd_config file has this explicitly set to "no".

c) PasswordAuthentication - you may want to change this to "no" and force users to setup a public/private key pair in order to login to the server.

(note: this post was never completed... so use with a grain of salt)

rsync.conf file for Cygwin environments

You should definitely refer to the official rsync website for the real documentation on configuring the rsyncd.conf file.

Locate your /etc folder under where you installed Cygwin. Since I installed Cygwin to C:\bin\cygwin, my /etc folder is C:\bin\cygwin\etc. For a fresh install, you'll need to create the "rsyncd.conf" file in that folder (C:\bin\cygwin\etc\rsyncd.conf).

(minimal rsyncd.conf file)

use chroot = false

strict modes = false

log file = rsyncd.log



[test]

path = /cygdrive/d/rsync/test

read only = false

transfer logging = yes

Minimal Cygwin install for RSync and SSH

Source links:

How to setup the secure shell daemon on a Windows 2000 machine?
Windows Rsync Server Setup
CygwinInstallationGuide (a wiki topic about the cygwin installation)

Note: The following probably doesn't work (probably missing a package, or the fact that I have GNU's unix tools for Win32 installed is problematic), but I might come back and make it work later so I'm leaving it here for now. I ran into trouble when trying to configure SSH. Right now, I've gone back to my original plan of either hacking apart the Cygwin files and manually copying only the DLLs and EXEs that I need or using the OpenSSH for Windows project at SourceForge.

1. Run the Cygwin setup.exe file and start the instllation. I chose to install to "c:\bin\cygwin", but left the rest of the options "as-is". Pick your mirror (use the Cygwin public mirrors page to find one close to you).

2. On the "Select Packages" screen, select the "Curr" option and make sure it says "Category" next to the "View" button at the top. The installation dialog is (finally) re-sizeable, so stretch it out or maximize it so you can see all of the columns.

3. Beside the "+All" category, it will say "Install", "Uninstall", ... click on the word until all of the categories say "Uninstall". (Note: These steps assume that you're doing a new Cygwin install and that you don't already have Cygwin installed.) Now we can start picking the minimum number of packages required to setup SSH and RSync.

4a. Under the "+Admin" category, you'll need to install the "cygrunsrv" package (click once on the "Skip" indicator under the "New" column). This will turn on a few other packages that this package depends on (mostly under the "+Base", "+Libs", and "+Shells" categories).

4b. Open up the "+Net" category and select the "rsync" and "openssh" packages. You'll also end up with "openssl" which is required in order to use "openssh".

5. Click the "Next" button to start downloading and installing the packages. If the download fails, choose another mirror, double-check your package selections (my copy remembered which packages I had already selected), and try again. The base install size required around 7MB of downloads and expanded out to 24MB (34MB actual due to a 4KB cluster size).

6. Fire up the cygwin shell, you should see a command-line window open with a "$" prompt. Try out a few unix commands (pwd, ls, whoami) to see if things are working.

7. Further steps... (I'll cover these in future posts)

a) Setup your rsync.conf file (in the "etc" folder)
b) create a service account for use by the rsync service
c) create a Windows service using the "cygrunsvc" tool
d) setup OpenSSH and then re-configure rsync to use it

Hacking together a minimal rsync for windows installation

Based on what I've read elsewhere (links in my previous posting), I think I can pull the relevant pieces out of the Cygwin package. I'll try to keep good notes as to what worked and what didn't, but let me know if you find any errors. Rsync wrapper for Win32 seems to be a good starting point for which DLLs and files I'll need to pull out of the standard Cygwin release.

You can download the files off of any of the Cygwin public mirrors. Grab the following archives and extract them to a temporary directory on your machine.

release/cygwin/cygwin-1.5.10-3.tar.bz2
- contains the DLL file (usr/bin/cygwin1.dll) and a lot of base utilities

release/popt/libpopt0/libpopt0-1.6.4-4.tar.bz2
- contains the usb/bin/cygpopt-0.dll file

release/rsync/rsync-2.6.2-1.tar.bz2
- RSync (rsync executable)

Create a folder where you're going to store the rsync files (I use C:\bin\rsync).

Copy the following files to your rsync folder:

cygwin1.dll

cygpopt-0.dll

rsync.exe

Create your rsync.conf file and put it in your rsync folder.

Test out whether you've gotten rsync working (thanks to "Aaron Johnson's page about rsync" for showing me what command line options to use). To do this, type the following commands:

c:

cd \bin\rsync

rsync --config="c:\bin\rsync\rsyncd.conf" --daemon

If you have a log file, there should now be an entry indicating that rsync has started up and is listening on the default port (tcp/873). Looking at the processes in Windows Task Manager, you should see the "rsync.exe" process. You should also now test out some rsync transfers from another workstation to verify that your security settings and module settings are correct.

To do:
- create the user account to use for the rsync service
- setup rsync to run as a service (need the SRVANY.EXE file, I think)
- figure out how to get rsync talking through an SSHD server

RSync and Windows

This is a follow-up to my previous post about Securing cwRSync. We were using the "cwRSync package", but when running in server mode it doesn't know how to talk to clients over an SSH-encrypted connection. Which isn't a big deal if you're only talking to other servers on the local network, but is problematic in cases where you have to be wary of eavesdropping (across WiFi links or untrusted networks like the internet). So I've been looking off-and-on over the past month at figuring out how to get an rsync service running using SSH on a Windows server.

One option is to install the full Cygwin package. Which is a bit much for a server (or rather, I'm not comfortable installing Cygwin on a server... yet).

Another option seems to be the OpenSSH for Windows project at SourceForge. That doesn't include rsync though, just scp. So I might look at "Installing ssh and rsync on a Windows machine: minimalist approach" which requires an absolute bare minimum of files to be installed. However, the files at that location are from Jan 2002, which is a bit old and the latest version as of July 2004 for the Cygwin DLL is cygwin-1.5.10-2.

Linux on Laptops

Today's useful link is: Linux on Laptops.

They have a short-n-sweet index where people submit links to their HOWTO pages regarding how to get a specific distro to work on a particular make/model of laptop.

Gentoo: Setting up PostgreSQL

Getting PostgreSQL installed really isn't that difficult on Gentoo Linux.

# emerge -s postgresql
# emerge postgresql
(install takes a while, didn't time it)
# ebuild /var/db/pkg/dev-db/postgresql-7.4.2-r1/postgresql-7.4.2-r1.ebuild config
(a few messages later)
 * The current value of SHMMAX is too low for postgresql to run.
 * Please edit /etc/sysctl.conf and set this value to at least 134217728.
 * 
 *   kernel.shmmax = 134217728
 * 

Fire up nano... see for an explanation of why we need to edit sysctl.conf. The short version is that the 2.6 linux kernel has a default value (shared memory limits) that is too small to be compatible with PostgreSQL.

# nano -w /etc/sysctl.conf

(add the following lines)

#Kernel parameters for PostgreSQL
#default is 32MB, PostGreSQL needs 128MB
kernel.shmmax = 134217728
kernel.shmall = 134217728

Now manually update the current values and start the server.

# echo 134217728 >/proc/sys/kernel/shmall
# echo 134217728 >/proc/sys/kernel/shmmax
# rc-update add postgresql default
# /etc/init.d/postgresql start

Now I'm off to explore the PostgreSQL documentation.

The default Gentoo install seems to already include a "postgres" user in /etc/passwd. To get logged in as the postgres user account, you will (I think) first need to switch to root.

# su
# cd /usr/local
# su - postgres

Now you can continue with section 16. Skip the page about creating the database cluster, it's already been created in "/var/lib/postgresql/data" back when you ran the "ebuild config" command. You can verify this by looking at the config file ("cat /etc/conf.d/postgresql"), where the PGDATA= line indicates the location of the database.

In fact, skip straight to chapter 16.4 - Run-time Configuration, because the server is already running. To verify that the server is running, "cat /var/lib/postgresql/data/postmaster.pid". Make a note of the PID on the first line (second line is the database location), then "cat /proc/nnnn/status" (replacing "nnnn" with the PID).

Misc Mozilla Bits

Just a few misc Mozilla 1.7 settings that I've found useful. All of these need to be added/changed in your prefs.js file in your profile directory. Make sure that you've exited out of all Mozilla windows, including the QuickLaunch icon in the system tray before making your edits. Otherwise, when Mozilla exits again later, it will overwrite your changes.

It's also a good idea to make a backup file of your prefs.js file prior to making changes.

1) Changing the trash folder in Mozilla Mail (or Thunderbird) to match what is used on your IMAP server. The standard trash folder is called "Trash", but my IMAP service uses "Deleted Items" instead. To make things simple, I changed Mozilla to use "Deleted Items" as well for that particular account. Replace "serverx" with the appropriate server number (e.g. "server7") that matches your IMAP account.

user_pref("mail.server.serverx.trash_folder_name", "Deleted Items");

Changing the default folder for saved copies of sent items from "Sent" to "Sent Items" is a bit easier. Just right-click on the e-mail account and pick "Properties", then look in "Copies & Folders" and change where Mozilla/Thunderbird stores copies of e-mails that you have sent.

For the technically minded, the lines in "prefs.js" that are affected by this change are:

user_pref("mail.identity.id5.fcc_folder", "imap://username@imap.somedomain.com/Sent Items");
user_pref("mail.identity.id5.fcc_folder_picker_mode", "1");

2) Displaying an error page instead of just a blank page when a webpage times out. One of the most annoying features in base Mozilla/Firebird is the way that they handle timeout errors. Instead of getting an error message on the screen, you get a blank page and the location bar will have been cleared. Which, if you were trying to load the page in the background for later viewing, means you have to try and remember or figure out what link you were trying to look at. Adding the following line to your prefs.js file will at least give you an error display that lets you retry the URL:

user_pref("browser.xul.error_pages.enabled", true);

Removable PATA/SATA Drive Bays

Shopping around for some removable drive bays (either PATA or SATA). I definitely don't want anything plastic, which cuts the field a bit. StarTech seem to make some nice drawers with multiple fans.

It's a pity that their multi-bay SATA systems aren't ready yet, because that looks interesting. (Holds 3 SATA drives in removable shells, takes up 2 5.25" drive bays, has a single large fan on the back.)

DRW115ATA
DRW115ATABK - Black, aluminim, PATA, ~$60
DRW115SAT
DRW115SATBK - SATA, aluminum, black, ~$60-70

Extra bays are $20-$30, extra caddies are $50-$60.

Since we'll be using these for backups, we'll probably go with PATA and 5400rpm drives (which run a lot cooler then 7200rpm SATA drives). I have to allow for ambient temperatures up to 85F at the office and I'm afraid that 7200rpm drives would cook themselves.

Update: I now own a set of the DRW115ATA drawers. The construction is quite sturdy, but the fit and finish is not always the best. They can be moderately difficult to slide in and out of the drive bays.

SPF Records

Published SPF records for my domain this week.

Linux Bare-Metal Backup and Recovery Planning

The whole concept behind bare-metal restore is that it's the shortest point from pulling a replacement set of server hardware out of a box to having a working system to replace the one that's down. Ideally, without having to go through re-installing the operating system (which, in the case of Gentoo, can take a few hours). It also (generally) only works when you have exactly the same hardware in the replacement box as the box that's kaput. So if you're using any exotic hardware (e.g. a fancy RAID card), you'd best buy (3). That way when the first one dies, you can put in one of the (2) spare cards and still get at your data.

Some backup software makes the process as simple as dropping a boot CD in the drive and inserting the most recent bare-metal backup (sometimes called a "gold" tape?) tape. It can also be a multi-step process, where the first step restores the operating system to a known state and the second step handles restoring the data and applications.

Since I'm not ready to tackle writing a guide, I'll settle for a few links:

Linux Complete Backup and Recovery HOWTO (by Charles Curley) - A very good starting point.

Linux Journal: Bare Metal Recovery, Revisited (by Charles Curley, Aug 2002) - A good sidebar discussion of the HOWTO document that he wrote.

About.com: Linux backup - A collection of links to backup software.

Unix SysAdm Resources: Backup & Archival Software for Unix (Stokely Consulting) - Listing of backup software for unix.

Linux Backups Mini-FAQ (Karsten M. Self) - Good article on simply using "tar".

Gentoo: Segmentation fault in vgscan during boot

Now for the other error that I got during the initial bootup.

* Using /etc/modules.autoload.d/kernel-2.6 as config:
* Loading module dm-mod...                                                 [ ok ]
* Autoloaded 1 module(s)
* Setting up the Logical Volume Manager...
/sbin/rc: line 429: 4422 Segmentation Fault /sbin/vgscan >/dev/nul         [ ok ]
* Starting up RAID devices: ...
* Checking all filesystems...
/dev/md0: clean, 39/18072 files, 5573/72192 blocks
fsck.ext: No such file or directory while trying to open /dev/vgmirror/opt
/dev/vgmirror/opt:
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 

("No such file or directory..." error repeats for all of the other 
logical volumes in the volume group(s) on the system)

My initial guess is that the software RAID is not loading up prior to the LVM stuff trying to load. Possibly, I'll have to edit the ordering in "/etc/init.d/checkfs", however since RAID is compiled into the kernel as built-in, and the LVM stuff is a module, the RAID should've already started prior to the LVM stuff.

Looking closer at the boot screen, I can see the "md:" lines correctly autodetecting the RAID arrays. So RAID support seems to be working fine. In fact, if I login to maintenance mode, and "cat /proc/mdstat", all of the RAID arrays show up correctly.

Attempt #1:

Moved things around in /etc/init.d/checkfs. No change in the end-result, except that the messages are re-ordered ("Starting up RAID devices" now appears before "Setting up the Logical Volume Manager") and the error message changes to "4422 Segmentation Fault /sbin/vgscan >/dev/nul". Probably a dry-hole in terms of finding and fixing the real problem.

Attempt #2:

Flipped back to the Gentoo LVM2 documents to see if I missed anything in setting up the LVM set to auto-mount at startup. Booted my way into maintenance mode and use "vgscan -v" to let vgscan attempt to find all of the volume groups. "vgscan" will take a while to run, at least with verbose (-v) mode you'll be able to see some status. On my setup, "vgscan" correctly located the "vgmirror" volume group.

Did a look at the "/etc/lvm" folder on the root volume using "ls -la /etc/lvm" and saw something surprising. There is a ".cache" file which is huge (mine was 10881785 in size). Doing a "cat" of the contents, I see some entries like "/dev/discs/disc2/discs/disc2.../disc2/md/254" which looks like a recursive loop of some sort.

Hint #2, run "vgdisplay -vv" and I see the error message "Too many levels of symbolic links" after each of those long entries. I also see this problem if I run "vgscan -vv". I finally changed my "/etc/lvm/lvm.conf" file to look like the following, and vgscan and vgdisplay are very quick at finding the volume group on my raid array and no longer segfault while looking at other items:

devices = {
    scan=["/dev/md"]
    filter=["a|^/dev/md/3$|","r/.*/"]
    }

Note that this filter only allows vgscan to scan the "md3" device. This keeps vgscan from scanning other devices that don't need to be scanned on my system (and fixes the segfault issue where it goes into infinite recursion on certain devices). If you need to scan other RAID devices (/dev/md1, etc.) or other physical partitions, then you'll need to adjust the "accept" portion of the filter.

Save, shutdown the raid (raidstop -a /dev/md0 for each /dev/md* device), and reboot. My server now boots up correctly.

Links:

Re: mkfs.xfs on software raid5 (2.6.5 kernel) - MD array /dev/md2 not in clean state (alt.os.linux.gentoo) - Shows the exact error message that I'm seeing.

Re: [gentoo-user] LVM2 Date: 2004-03-31 15:40:13 PST (linux.gentoo.user) - Talks about the ordering in which software RAID and the LVM modules load.

Example of lvm.conf file - shows a more complex lvm.conf file, complete with multiple filters.

A more complete example lvm.conf file

Gentoo: Failed to load mii

More self-inflicted pain (guarantee that I'm doing this to myself, not due to the Gentoo install guide...)

During boot-up after installing the 2.6.6 kernel, the following (2) modules fail to load:

Loading module mii...
Failed to load mii

Loading module via-rhine...
Failed to load via-rhine

The second error is because I had configured the kernel to load "via-rhine" as built-in and not as a module. You can do one or the other, but not both.

Not sure about the "mii" error, I'll merely comment it out in the autoload config file for now and see what happens.

Securing cwRSync

At the office we're working on setting up cwRSync on the web server array to push the daily web/ftp/smtp log files back to a central point for archiving. Right now, since all of the web servers are on the same LAN segment at the hosting facility, we're just sending the plain text data across the wire to the rsync port (tcp/873). Since the previous solution was to use FTP to move the log files around, it's no worse then the old solution from a security standpoint. (It is, however, much faster and more efficient.) Security is handled solely thorugh the rsyncd.conf "hosts allow" setting (only the internal IP addresses are allowed to be used to transfer the data) with no passwords or shared keys.

However, since the next step is that we want to setup pulling those log files automatically back to the main office, we need to look into locking it down further and putting encryption in place (e.g. routing rsync traffic over an ssh tunnel).

After digging around a bit here's what I've found:

The cwRSync Service does not support SSH, so there's no way to connect securely to a rsync server that is using cwRSync as its daemon. Future releases are expected to add ssh support for cwRSync servers. Locking down through IP address and username/password is the limit of what you can do for security, all traffic is in the clear (unless you have IPSec between the two machines).

However, you can use cwRSync in a client-configuration and route the traffic over SSH to a SSH-capable rsync server.

That being said, I'm going to explore some other packages. All of which will either require that cygwin be installed, or at least that certain cygwin DLLs be installed.

Links:

Rsync wrapper for Win32 - Uses the cygwin DLLs, but doesn't require a full cygwin install, includes SSH.

Troubleshooting software RAID boot problems

First problem is that the system boots straight into "grub". Probably due to a missing "grub.conf" file, which I'm pretty sure I had written to the proper location earlier in the install. So I'll load the kernel by hand and fix the config file once I get the box to boot properly.

grub>
grub> cat /boot/grub/grub.conf
(no file found)

grub> root (hd0,0)
grub> kernel /kernel-2.6.6-gentoo root=/dev/md2
grub> boot

This starts the kernel boot process, which will then lead me to my second error. In the meantime, if you use [shift-PgUp] and [shift-PgDn] you can scroll back and forth through the boot messages. A minor problem is that since the raid didn't shutdown cleanly, there are numerous "md: mdx: raid array is not clean -- starting background reconstruction" messages. And I can't even begin to troubleshoot the "Kernel panic: No init found" error until resync is done (that's a 2.5-3.0 hour process).

Looking back at my kernel configuration, I see that Software RAID with RAID1 support was compiled as BUILTIN, and ext2/ext3 were installed by default if I remember correctly. Those are two of the possible errors that could cause the kernel not to be able to read from the "/" (root) partition.

Also possible is that the "/" filesystem was not properly mounted during the install. The following is what I've tried to do in order to fix the issue (or at least diagnose the issue). I would strongly recommend that you do not use the following on a production system unless you understand what everything does. Since I don't have any data on the system (other then config files), I have a good amount of latitude with regards to what I can do. Hopefully, since /boot, /usr, /opt, /var and /home are intact, things will go quicker then the first install.

This repair process is basically a complete reinstall because nothing else under "/" (root) actually got written to the hard drive. Only the separately mounted folders such as /boot, /opt, /usr, /var, and /home were properly mounted.

Note: The following steps assume that you have nothing on your partitions worth keeping, or that you've already backed everything up. I tried doing it without formatting the "/" partition, but the bootstrap.sh file keeps dying on line 84. So I'm going to do a format of the "/" partition as well as /opt, /usr, /var, /tmp, and /var/tmp.

Boot the LiveCD, at the boot prompt be sure to pick the boot kernel and pass the arguments that you want, then load the raid and LVM modules.

boot: gentoo -nohotplug
(gentoo kernel now loads)

livecd root # modprobe md
livecd root # modprobe dm-mod
livecd root # modprobe via-rhine    (if your network adapter failed to autoload)
livecd root # net-setup eth0        (if your network adapter failed to autoload)

If you have a copy of your /etc/raidtab file on floppy, copy it in now, otherwise you'll have to re-key your /etc/raidtab by hand. If you need, you can temporarly mount partitions to copy files from.

livecd root # nano -w /etc/raidtab
livecd root # raidstart -a
livecd root # cat /proc/mdstat
(verify that all raid sets are up and running)

livecd root # swapon /dev/md1
livecd root # mke2fs -j /dev/md2     (OVERWRITES YOUR / PARTITION)
livecd root # mount /dev/md2 /mnt/gentoo
livecd root # mkdir /mnt/gentoo/boot
livecd root # mount /dev/md0 /mnt/gentoo/boot
livecd root # ls /mnt/gentoo/boot
(if you have files already in /boot, this verifies that you mounted in the proper order)

livecd root # mkdir /etc/lvm
livecd root # echo 'devices { filter=["r/cdrom/"] }' > /etc/lvm/lvm.conf
livecd root # vgscan
(verify that it found your existing volume group or groups)

livecd root # vgchange -ay vgmirror
livecd root # lvscan
(verify that your logical volumes now show up and are "ACTIVE")

livecd root # mke2fs -j /dev/vgmirror/opt    (OVERWRITES YOUR / PARTITION)
livecd root # mke2fs -j /dev/vgmirror/usr    (OVERWRITES YOUR / PARTITION)
livecd root # mke2fs -j /dev/vgmirror/var    (OVERWRITES YOUR / PARTITION)
livecd root # mke2fs -j /dev/vgmirror/home   (OVERWRITES YOUR / PARTITION)
livecd root # mke2fs /dev/vgmirror/ttmp      (OVERWRITES YOUR / PARTITION)
livecd root # mke2fs /dev/vgmirror/vartmp    (OVERWRITES YOUR / PARTITION)

livecd root # mkdir /mnt/gentoo/opt
livecd root # mkdir /mnt/gentoo/usr
livecd root # mkdir /mnt/gentoo/var
livecd root # mkdir /mnt/gentoo/home
livecd root # mount /dev/vgmirror/opt /mnt/gentoo/opt
livecd root # mount /dev/vgmirror/usr /mnt/gentoo/usr
livecd root # mount /dev/vgmirror/var /mnt/gentoo/var
livecd root # mount /dev/vgmirror/home /mnt/gentoo/home

livecd root # mkdir /mnt/gentoo/tmp
livecd root # mount /dev/vgmirror/tmp /mnt/gentoo/tmp
livecd root # chmod 1777 /mnt/gentoo/tmp
livecd root # mkdir /mnt/gentoo/var/tmp
livecd root # mount /dev/vgmirror/vartmp /mnt/gentoo/var/tmp
livecd root # chmod 1777 /mnt/gentoo/var/tmp
livecd root # mkdir /mnt/gentoo/proc
livecd root # mount -t proc none /mnt/gentoo/proc

livecd root # date
livecd root # cd /mnt/gentoo
livecd gentoo # tar -xvjpf /mnt/cdrom/stages/stage1-x86-20040218.tar.bz2
livecd gentoo # tar -xvjf /mnt/cdrom/snapshots/portage-20040223.tar.bz2 -C /mnt/gentoo/usr
livecd gentoo # mkdir /mnt/gentoo/usr/portage/distfiles
livecd gentoo # cp /mnt/cdrom/distfiles/* /mnt/gentoo/usr/portage/distfiles/

livecd gentoo # nano -w /mnt/gentoo/etc/make.conf
(repeat your make.conf from the initial install)

livecd gentoo # mirrorselect -a -s4 -o | grep -ve '^Netselect' >> /mnt/gentoo/etc/make.conf
livecd gentoo # cp -L /mnt/gentoo/etc/make.conf /mnt/gentoo/boot/make.conf-backupcopy
livecd gentoo # cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
livecd gentoo # cp -L /etc/raidtab /mnt/gentoo/etc/raidtab
livecd gentoo # cp -L /etc/raidtab /mnt/gentoo/boot/raidtab-backupcopy
livecd gentoo # mkdir /mnt/gentoo/etc/lvm
livecd gentoo # cp -L /etc/lvm/lvm.conf /mnt/gentoo/etc/lvm/lvm.conf
livecd gentoo # cp -L /etc/lvm/lvm.conf /mnt/gentoo/boot/lvm.conf-backupcopy
livecd gentoo # chroot /mnt/gentoo /bin/bash
livecd / # env-update
livecd / # source /etc/profile
livecd / # emerge sync
livecd / # cd /usr/portage
livecd / # scripts/bootstrap.sh

If bootstrap runs correctly (and it should now that I re-formatted the /opt, /usr, /var, /home, /tmp, and /var/tmp volumes), I can pick back up with the rest of my original install process