Linux KVM shutting down all virtual guests

On my current virtualization server running Linux KVM (QEMU), I want to shutdown all guests so that I can unmount the file system containing the VM image files and make it larger.

#1 - See what guests are running

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     dc1                            running
 3     cfmc87                         running
 4     win7c                          running

#2 - Use the libvirt-guests service to suspend all of them.

# service libvirt-guests stop

Running guests on default URI: dc1, cfmc87, win7c

Suspending guests on default URI...
Suspending dc1: ...
Suspending dc1: done
Suspending cfmc87: ...
Suspending cfmc87: ...
Suspending cfmc87: done
Suspending win7c: ...
error: Failed to save domain bb0d169d-a373-544e-a0f7-99e338673177 state
error: internal error unable to execute QEMU command 'migrate': An undefined error has ocurred

#3 Dealing with troublesome guests

Oops, looks like win7c is being difficult.  So let's just force it down.

# virsh shutdown win7c
Domain win7c is being shutdown
# virsh list      
 Id    Name                           State
----------------------------------------------------
 4     win7c                          running

Even with issuing the shutdown command and waiting a few minutes, the "win7c" guest is still running.  So we'll have to "destroy" the instance.

# virsh destroy win7c 
Domain win7c destroyed

#4 Checking that all files are released

Now I can use the "lsof" command to verify that there are no open files on the mount point.

# lsof /srv/vms
(returns nothing)

#5 Umount, fsck, resize, fsck, remount

The VM image LV is stored on /dev/md127, inside a LVM thin pool.  It's currently 200GB and is about 70% full, so I want to add another 100GB.

# umount /srv/vms
# fsck -f /dev/vg10/vms
# lvextend -L+100G /dev/vg10/vms
# resize2fs /dev/vg10/vms
# fsck -f /dev/vg10/vms

# mount /srv/vms

#6 Restart the guests

# service libvirt-guests restart

# virsh list

And now my /srv/vms filesystem is no longer having space issues.

First impressions of ext4 with large file deletion

I've been using ext3 for a long time on my CentOS servers, but ext3 has one big flaw that has constantly affected me.  It takes forever and a day in order to delete large files.

On my home server, I have a very large array where I put raw video files prior to conversion into compressed/lossy formats.  This works well, but then when I go to clean off the multi-gigabyte raw files it will take 5-10 minutes to delete a few dozen gigabytes of data.  So I've been putting up with for a long time this way.

Fast forward to 2010, and ext4 is finally close to production ready in CentOS 5.5.  (It may even already be marked as production-ready...)  I switched that scratch space over to ext4 last week and it's performing admirably.  Much better performance when deleting large multi-gigabyte files.  Which helps the system not feel so slow while it's doing that delete.

I probably won't use ext4 for my primary volumes quite yet.  I still plan on leaving /home, / (root), /var, /boot as ext3 (ext2 for /boot), but I will probably put the larger user data file systems as ext4 from now on.  It's enough of a step forward to be worth it (and ext4 is pretty stable now).