Ubuntu and software RAID, getting a device path that won't change from boot to boot

While I love mdadm (software RAID), it's perplexing me at the moment as it keeps changing its device number under Ubuntu.  When I created the array, I created it as "md100", but whenever I restart it ends up as "md127" (and could end up as something else!).  Normally, this doesn't matter, but I'm doing LVM on LUKS, so I need a static (unchanging) path to the array device.

This is a (4) disk array, running mdadm's raid10, and was created with the command:

$ sudo mdadm --create /dev/md100 --raid-devices=3 --spare-devices=1 --level=raid10 /dev/sd[bcde]1

After creating the array, I can check the details with:

$ sudo mdadm --detail --scan /dev/md100

ARRAY /dev/md127 metadata=1.2 spares=1 name=freya:100 UUID=deafbeef:deadbeef:beafdeff:beaffffa

Notice the "name=freya:100".  That's the key to finding a static path to the array.  If I then look under the /dev/md directory, I will see:

$ ls -l /dev/md
total 0
lrwxrwxrwx 1 root root 8 Dec 30 08:15 freya:100 -> ../md127

That means the static path to this array is "/dev/md/freya\:100" and I can use LUKS format on it with:

$ sudo cryptsetup -y -v luksFormat /dev/md/freya\:100

Alternate, I can search for the UUID in the /dev directory and find:

$ sudo find /dev -name '*deafbeef*'

/dev/disk/by-id/md-uuid-deafbeef:deadbeef:beafdeff:beaffffa

I can then add a LUKS keyfile to the device and unlock the device at boot by listing it in the /etc/crypttab file.  Either path will work, but the colons (:) will likely have to be escaped in /etc/crypttab.

PS: Yes I've tried putting the array line in the /etc/mdadm/mdadm.conf file as just "ARRAY /dev/md100 UUID=deafbeef:deadbeef:beafdeff:beaffffa", which is supposed to fix the issue.

PS #2: It's interesting that the mdadm UUID appears in /dev/disk/by-id and not /dev/disk/by-uuid.

Linux KVM shutting down all virtual guests

On my current virtualization server running Linux KVM (QEMU), I want to shutdown all guests so that I can unmount the file system containing the VM image files and make it larger.

#1 - See what guests are running

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     dc1                            running
 3     cfmc87                         running
 4     win7c                          running

#2 - Use the libvirt-guests service to suspend all of them.

# service libvirt-guests stop

Running guests on default URI: dc1, cfmc87, win7c

Suspending guests on default URI...
Suspending dc1: ...
Suspending dc1: done
Suspending cfmc87: ...
Suspending cfmc87: ...
Suspending cfmc87: done
Suspending win7c: ...
error: Failed to save domain bb0d169d-a373-544e-a0f7-99e338673177 state
error: internal error unable to execute QEMU command 'migrate': An undefined error has ocurred

#3 Dealing with troublesome guests

Oops, looks like win7c is being difficult.  So let's just force it down.

# virsh shutdown win7c
Domain win7c is being shutdown
# virsh list      
 Id    Name                           State
----------------------------------------------------
 4     win7c                          running

Even with issuing the shutdown command and waiting a few minutes, the "win7c" guest is still running.  So we'll have to "destroy" the instance.

# virsh destroy win7c 
Domain win7c destroyed

#4 Checking that all files are released

Now I can use the "lsof" command to verify that there are no open files on the mount point.

# lsof /srv/vms
(returns nothing)

#5 Umount, fsck, resize, fsck, remount

The VM image LV is stored on /dev/md127, inside a LVM thin pool.  It's currently 200GB and is about 70% full, so I want to add another 100GB.

# umount /srv/vms
# fsck -f /dev/vg10/vms
# lvextend -L+100G /dev/vg10/vms
# resize2fs /dev/vg10/vms
# fsck -f /dev/vg10/vms

# mount /srv/vms

#6 Restart the guests

# service libvirt-guests restart

# virsh list

And now my /srv/vms filesystem is no longer having space issues.

Linux server partitioning with mdadm software RAID and LVM2

Over the years, I've really come to appreciate what judicious use of LVM (or LVM2) brings to the table when administering servers. If you rely on it heavily and leverage it properly, you can do things like:

• Snapshot any file system (other then /boot) for backups, or to make images, or to test something out.
• Migrate a logical volume (LV) from one physical volume (PV) to another, without having to take the file system offline or deal with downtime.
• Resize file systems that are too large or too small, with minimal downtime (if the file system supports it).

Basically, other then /boot, if you're thinking of creating a separate partition or Software RAID device the you should be using LVM instead of physical partitions or RAID devices. You gain a lot of flexibility in the long-run and setting up LVM on top of hardware or software RAID or plain old disks is no longer that difficult.

These days, when I setup disk partitions to hold a server's boot-up files, I only create (2) partitions on the drive. One for the Software RAID-1 mirror set to hold /boot (usually 256-1024MB) and the rest of the drive is a second RAID-1 mirror set that is turned into a LVM physical volume (PV) and assigned to a volume group. I will usually only partition out to about 99% of the drive size if I'm doing Software RAID, because that makes it easier later to put in a different model disk of the same capacity and still have things work. Drives from different manufacturers have slightly different capacities, so you can run into trouble down the road when you go to replace a failed drive if you assumed all drives were exactly the size as your original drives.

Inside that LVM volume group (VG), I create all of my other partitions. These days, that means:

• / - the "root" partition, usually 16-32GB for CentOS 6
• /home - Usually starts at 4GB for a server where people won't be logging in much.
• /opt - 4-24GB
• /srv - 1-4GB (sub-directories get their own LV later)
• /tmp - 4-24GB
• /usr/local - 8-24GB
• /var - 4-24GB
• /var/log - 4-24GB
• /var/spool - 2-4GB to start
• /var/tmp - 2-8GB to start

And that's just the basic operating system file systems. For things like squid, e-mail, web servers, samba shares, etc., each of those will get its own LV, allocated from the server-wide volume group.

Follow ups:

• GRUB2 understands mdadm (software RAID) and LVM. So we will eventually be able to put /boot in an LVM volume. But the GRUB that ships with RHEL6 and CentOS6 is still GRUB 0.97.

LUKS: /dev/mapper "read failed after 0 of 4096 at 0: Input/output error"

We're using external USB drives for our backups, protected using LUKS/cryptsetup. On our 3-4 year old Opteron 2210 HE CPU running at 1.8GHz, we estimate that LUKS can perform about 60-70 MB/s per CPU core. We mount the LUKS volumes automatically (at server boot) by listing them in /etc/cryptsetup and using a key-file instead of having to enter a password and autofs handles the automatic mounting/dismounting of the ext4 file system inside the LUKS volume.

It all works very well, until you remove the USB device and then run pvscan/lvscan commands of LVM. Which then throws the following errors:

# pvscan
  /dev/mapper/USBOFFSITE12B: read failed after 0 of 4096 at 999201636352: Input/output error
  /dev/mapper/USBOFFSITE12B: read failed after 0 of 4096 at 999201693696: Input/output error
  /dev/mapper/USBOFFSITE12B: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/USBOFFSITE12B: read failed after 0 of 4096 at 4096: Input/output error
  PV /dev/md12   VG vg13   lvm2 [2.67 TiB / 821.62 GiB free]
  Total: 1 [2.67 TiB] / in use: 1 [2.67 TiB] / in no VG: 0 [0   ]

If 'autofs' would allow us to execute a script after it dismounts the drive from inactivity, it might be a good option for us. But I'm not sure that is possible.

Another option would be that at the end of the daily backup script which copies files off to the attached device, we dismount it automatically.

A third option would be to check every hour and see whether the /dev/mapper/NAME is no longer in use, then tell cryptsetup to dismount it. The command to check that might be "dmsetup ls --tree -o uuid,open | grep -i 'CRYPT-LUKS'".

Still exploring options at this point. I need to do some more testing first. I'm also searching for a way to auto-open LUKS volumes upon device insertion.

Migrate from Direct Partitions to LVM Volumes

Weekend Project: Migrate from Direct Partitions to LVM Volumes by Nathan Willis

This tutorial covers the basics of migrating from regular disk partitions to LVM logical volumes (LVs). It displays some of the strengths of LVM, where you can resize logical volumes, or move logical volumes to another physical device in the same volume group.

LVM Maximum number of Physical Extents

Working on a 15-disk system (750GB SATA drives) so this issue came up again:

What is the maximum number of physical extents in LVM?

The answer is that there's no limit on the number of PEs within a Physical Volume (PV) or Volume Group (VG). The limitation is, instead, the maximum number of PEs that can be formed into a Logical Volume (LV). So a 32MB PE size allows for LVs up to 2TB in size.

Note: Older LVM implementations may only allow a maximum of 65k PEs across the entire PV. The VG can be composed of multiple PVs however.

So if you want to be safe, make your PE size large enough that you only end up with less then 65k PEs. Just remember that all PVs within a VG need to use the same PE size. So if you're planning for lots of expansion down the road, with very large PVs to be added later, you may wish to bump up your PE size by a factor of 4 or 8.

A good example of this is a Linux box using Software RAID across multiple 250GB disks. The plan down the road is to replace those disks with larger models, create new Software RAID arrays across the extended areas on the larger disks, then extend VGs across new PVs. At the start, you might only have a net space (let's say RAID6, 2 hot-spares, 15 total disks) of around 2.4TB. That's small enough that a safe PE size might be 64MB (about 39,000 PEs).

Except that down the road, disks are getting much larger (1.5TB drives are now easily obtainable). So if we had stuck with a 64MB PE size, our individual LVs could be no larger then 4TB. If we were to put in 2TB disks (net space of about 20TB), the number of PEs would end up growing by about 8x (312,000). We might even see 4TB drives in a 3.5" size, which would be closer to 40TB of net space.

A PE size of 256MB might have served us better when we setup that original PV area. It would allow individual LVs sized up to 16TB. The only downside is that you won't want to create LVs smaller then 256MB and you'll want to make sure all LVs are multiples of 256MB.

Bottom line, when setting up your PE sizes, plan for a 4x or 8x growth.

Example:

The following shows how to create a volume group (VG) with 512MB physical extents instead of the default 4MB extent size (also known as "PE size" in the vgdisplay command output).

# vgcreate -s 512M vg6 /dev/md10

References:

LVM Manpage - Talks about the limit in IA32.
Maximum Size Of A Logical Volume In LVM - Walker News (blog entry)
Manage Linux Storage with LVM and Smartctl - Enterprise Networking Planet

LVM and SELinux

I was a bit perplexed... I had created a LV called /dev/vg/svn, had it mounted, was reading/writing data to it with no issues. But after I rebooted the CentOS5 server, I'm unable to mount the LV.

[root@localhost /]# /usr/sbin/pvscan
PV /dev/md6   VG vg   lvm2 [144.78 GB / 59.78 GB free]
Total: 1 [144.78 GB] / in use: 1 [144.78 GB] / in no VG: 0 [0   ]
[root@localhost /]# /usr/sbin/vgscan
Reading all physical volumes.  This may take a while...
Found volume group "vg" using metadata type lvm2
[root@localhost /]# /usr/sbin/lvscan
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay vg
--- Logical volume ---
LV Name                /dev/vg/svn
VG Name                vg
LV UUID                taYjia-BWWs-IWG3-313k-VoC2-ghik-01mFCg
LV Write Access        read/write
LV Status              NOT available
LV Size                85.00 GB
Current LE             21760
Segments               1
Allocation             inherit
Read ahead sectors     0

[root@localhost /]# 

So lvdisplay knows that the LV is there, but only if I tell it to look at the VG named "vg".

...

Turns out that it's an SELinux issue. Because SELinux was blocking access to the /etc/lvm/.cache file, it was causing problems. Fixing it was as simple as:

# cd /etc/lvm
# /sbin/restorecon -v .cache
# /usr/sbin/lvscan
inactive          '/dev/vg/svn' [85.00 GB] inherit

Gentoo: Segmentation fault in vgscan during boot

Now for the other error that I got during the initial bootup.

* Using /etc/modules.autoload.d/kernel-2.6 as config:
* Loading module dm-mod...                                                 [ ok ]
* Autoloaded 1 module(s)
* Setting up the Logical Volume Manager...
/sbin/rc: line 429: 4422 Segmentation Fault /sbin/vgscan >/dev/nul         [ ok ]
* Starting up RAID devices: ...
* Checking all filesystems...
/dev/md0: clean, 39/18072 files, 5573/72192 blocks
fsck.ext: No such file or directory while trying to open /dev/vgmirror/opt
/dev/vgmirror/opt:
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 

("No such file or directory..." error repeats for all of the other 
logical volumes in the volume group(s) on the system)

My initial guess is that the software RAID is not loading up prior to the LVM stuff trying to load. Possibly, I'll have to edit the ordering in "/etc/init.d/checkfs", however since RAID is compiled into the kernel as built-in, and the LVM stuff is a module, the RAID should've already started prior to the LVM stuff.

Looking closer at the boot screen, I can see the "md:" lines correctly autodetecting the RAID arrays. So RAID support seems to be working fine. In fact, if I login to maintenance mode, and "cat /proc/mdstat", all of the RAID arrays show up correctly.

Attempt #1:

Moved things around in /etc/init.d/checkfs. No change in the end-result, except that the messages are re-ordered ("Starting up RAID devices" now appears before "Setting up the Logical Volume Manager") and the error message changes to "4422 Segmentation Fault /sbin/vgscan >/dev/nul". Probably a dry-hole in terms of finding and fixing the real problem.

Attempt #2:

Flipped back to the Gentoo LVM2 documents to see if I missed anything in setting up the LVM set to auto-mount at startup. Booted my way into maintenance mode and use "vgscan -v" to let vgscan attempt to find all of the volume groups. "vgscan" will take a while to run, at least with verbose (-v) mode you'll be able to see some status. On my setup, "vgscan" correctly located the "vgmirror" volume group.

Did a look at the "/etc/lvm" folder on the root volume using "ls -la /etc/lvm" and saw something surprising. There is a ".cache" file which is huge (mine was 10881785 in size). Doing a "cat" of the contents, I see some entries like "/dev/discs/disc2/discs/disc2.../disc2/md/254" which looks like a recursive loop of some sort.

Hint #2, run "vgdisplay -vv" and I see the error message "Too many levels of symbolic links" after each of those long entries. I also see this problem if I run "vgscan -vv". I finally changed my "/etc/lvm/lvm.conf" file to look like the following, and vgscan and vgdisplay are very quick at finding the volume group on my raid array and no longer segfault while looking at other items:

devices = {
    scan=["/dev/md"]
    filter=["a|^/dev/md/3$|","r/.*/"]
    }

Note that this filter only allows vgscan to scan the "md3" device. This keeps vgscan from scanning other devices that don't need to be scanned on my system (and fixes the segfault issue where it goes into infinite recursion on certain devices). If you need to scan other RAID devices (/dev/md1, etc.) or other physical partitions, then you'll need to adjust the "accept" portion of the filter.

Save, shutdown the raid (raidstop -a /dev/md0 for each /dev/md* device), and reboot. My server now boots up correctly.

Links:

Re: mkfs.xfs on software raid5 (2.6.5 kernel) - MD array /dev/md2 not in clean state (alt.os.linux.gentoo) - Shows the exact error message that I'm seeing.

Re: [gentoo-user] LVM2 Date: 2004-03-31 15:40:13 PST (linux.gentoo.user) - Talks about the ordering in which software RAID and the LVM modules load.

Example of lvm.conf file - shows a more complex lvm.conf file, complete with multiple filters.

A more complete example lvm.conf file

LVM and Software RAID

Links:

Linux Logical Volume Manager (LVM) on Software RAID - explains the benefits of using LVM, and also how to get LVM running on top of software RAID in RedHat 8.

Google search for how to do software RAID and LVM

The Software-RAID HOWTO - Updated June 2004, so it's not an old outdated document. Also see the official URL.

Gentoo Server Project Wiki

Notes on Building a Linux Storage Server

Gentoo Forums: How to do a gentoo install on a software RAID - Thread was started back in July 2002, but there are posts as recently as June 2004.

Summary:

The "grub" boot loader needs a real physical partition to boot from (or possibly a RAID1 setup, RAID0 definitely doesn't work). What some folks do is to create (2) identical physical partitions on the drives in the RAID array, and then periodically copy from the primary partition to the secondary partition. (See gentoo forum post or search for user "wrex". Also look for posts by "hover" or "blake121666") The copy command is as simple as "dd if=/dev/hda1 of=/dev/hdb1 bs=8192b". You need to customize that to match your configuration (e.g. changing the source/dest partitions and the block size). If I can puzzle it all out, I think I'm going to go with /boot on RAID1. See section 7.3 of the Software RAID HOWTO

The "swap" partition should be mirrored. Otherwise when the disk fails, existing applications with data in the swap partition of the failed drive will die a nasty death. See section 2.3 of the Software RAID HOWTO.

There's a special set of command that need to be done in "grub" if you want to be able to pull the primary disk and still have the system bootable. (In case the primary disk would fail.) This is all mentioned in the gentoo forums thread about software RAID, look for the post by "hover". Alternatively, you can simply use a boot floppy and fixup the grub settings after the primary drive fails.

Replacing a failed disk in a software RAID1 array requires partitioning the drive by hand prior to rebuilding the array. This is also mentioned in the gentoo forums thread about software RAID, look for the post by "hover".

Searching around on my Gentoo 2004.0 Universal CD, I don't see the "mdadm" tools anywhere, but I do see "raidtools".

Gentoo LVM2 stuff

vgscan - displays the list of volume groups allocated on the system (for my box, I have vgos, vguser, vgtmp and vgmedia)
lvscan - displays the list of virtual partitions inside of the volume groups (for my box, I have 6)
vgdisplay - displays a lot of data bout the volume groups, a good place to find out how much space is free within a particular volume group (vgos has 12GB free, vguser 19GB free, vgtmp 7GB free, vgmedia 92GB free).

Back when I setup the box, I never created any partitions (volumes?) inside the vgmedia volume group. So now I want to think about what sort of media I'm going to be storing, and how to seperate it. Since most audio/video files are larger, I may consider using a larger block size (if that makes sense?). I'm also going to leave a small amount of space for backup files to be written to from the primary drive. So for now, I'll only allocate 32GB out of the 92GB.

lvcreate -L32G -nmedia vgmedia
mke2fs -j -c /dev/vgmedia/media
(then use an editor to add an entry to the /etc/fstab table)
mkdir /media
mount /dev/vgmedia/media /media

Now I'm ready to configure Samba. See the gentoo documentation about Samba.