Daemon News '200502' : '"FreeBSD System Disk Mirroring "'

DæmonNews: News and views for the BSD community

FreeBSD System Disk Mirroring

by Ralf S. Engelschall

Problem

RAID-1 (mirroring) is a popular approach to protect systems from a hard disk failure. It is done either in hardware or software. Using a hardware solution is common, and involves buying a RAID disk controller like one of the popular 3ware ATA RAID controllers. This avoids complications from software incompatibilities because the RAID array is presented to the system as one large physical disk.

The software solution is less expensive and more flexible, but it is important to be sure that booting directly from the resulting mirror setup is possible without ugly BIOS or boot loader tricks. Additionally, in the case of a major problem with the RAID-1 software driver, it is important to be able to rescue-boot from the mirror setup by treating it like a plain-disk setup again.

Possibilities

For software solutions in FreeBSD 5 one can choose between the raw ata(4), the good old ccd(4), the classic vinum(4), the new GEOM Vinum, and the GEOM gmirror(8).

Each one has its pros and cons:

ATA

Although the ATA driver's RAID functionality is intended primarily for hardware RAID controllers, it actually allows for creating an ATA RAID-1 on disks with any controller. This is the rawest approach. It is suitable only for ATA/IDE drives and has functional restrictions.
CCD

This is the good old Concatenated Disk Driver (CCD) which provides concatenation, RAID-0 (striping) and RAID-1 (mirroring). It was converted to use GEOM in FreeBSD 5, but it has fewer features than GEOM mirror.
Classic Vinum

This is the classic Vinum which provides all major RAID levels including RAID-1 and additionally is a full-scale volume manager. Although still available in FreeBSD 5, it does not use GEOM and panics the system when automatically enabled during the system boot. It is no longer feasible to use in FreeBSD 5, especially not for mirroring the system disks.
GEOM Vinum

This is the new GEOM Vinum, a functional port and code rewrite of Vinum as a GEOM class. It already provides lots of functionality but is still undocumented and hence not really ready for production use in FreeBSD 5.
GEOM Mirror

This is the GEOM Mirror, a dedicated GEOM class for establishing a RAID-1 array. It is a full-featured, from-scratch implementation and fully ready for production use, at least in FreeBSD 5.3-STABLE as of January 2005.

I currently prefer GEOM mirror for establishing a RAID-1 solution for the system disk partitions because it provides the ability to directly boot from the mirror without tricks, and is a really clean and modern solution. Additionally, GEOM-based solutions can be stacked in an arbitrary way. In other words, the resulting GEOM mirror is just another GEOM "provider" on which one can base any other GEOM "consumers", including GEOM striping (gstripe) or GEOM based disk encryption (gbde).

Solution

The FreeBSD 5 geom(4) framework is an excellent abstraction layer for disk I/O. The GEOM mirror class is a software RAID-1 implementation which can be used even to establish a mirror for the system disk partitions. It can be established for any GEOM "providers", including whole disks or just particular slices on a disk. Unfortunately each approach has pros and cons and hence each should be fully understood before use:

GEOM mirror on whole disk

This is straight-forward, obvious and results in having the whole disk mirrored, including its MBR. Unfortunately, it works only if the two disks are really equal in size or, at least, requires that the mirror is established on the smaller disk first and that extra space is lost on the second disk. In the case of a larger replacement disk (which happens often since newer disks tend to store more and cost less), one wastes even more disk space.
GEOM mirror on slices of a disk

This is a little bit less obvious, but provides a lot more flexibility and allows the usage of disks of different sizes and the possibility to make use of the disks' remaining space (either for a second mirror or as plain partitions).

GEOM mirror Approach 1: Whole Disk, Acceptable, Less Flexible

The following is a step-by-step command list for remotely converting a production FreeBSD 5.3-STABLE system from a single-disk (ad0) setup to a two-disk (ad0 & ad1) GEOM mirror (gm0) setup without the need for console access or Fixit/LiveFS CDROM. The assumption is that the two disks are equal in size or that the second disk is a little bit smaller than the first one.

# make sure the second disk is treated as a really fresh one
# (not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad1 bs=512 count=79

# place a GEOM mirror label onto second disk
# (actually on the last block of the disk)
gmirror label -v -n -b round-robin gm0 /dev/ad1

# activate GEOM mirror kernel layer
# (makes the /dev/mirror/gm0 device available)
gmirror load

# place a PC MBR onto the second disk
# (with a single FreeBSD slice /dev/mirror/gm0s1 covering the whole disk)
fdisk -v -B -I /dev/mirror/gm0

# place a BSD disklabel onto /dev/mirror/gm0s1
# (ATTENTION: in FreeBSD 5-STABLE before 14-Jan-2005 the
# /dev/mirror/gm0s1 device has to be specified as just "mirror/gm0s1" or
# the bsdlabel(8) will use the incorrect GEOM name "gm0s1" instead!)
# (NOTICE: figure out what partitions you want with "bsdlabel /dev/ad0" before)
# (NOTICE: start "a" partition at offset 16, "c" partition at offset 0)
bsdlabel -w -B /dev/mirror/gm0s1 # initialize
bsdlabel -e /dev/mirror/gm0s1    # create custom partitions

# manually copy filesystem data from first to to second disk
# (same procedure for partitions "g", etc)
newfs -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
dump -L -0 -f- / | (cd /mnt; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1d
mount /dev/mirror/gm0s1d /mnt/var
dump -L -0 -f- /var | (cd /mnt/var; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1e
mount /dev/mirror/gm0s1e /mnt/usr
dump -L -0 -f- /usr | (cd /mnt/usr; restore -r -v -f-)

# adjust new system configuration for GEOM mirror based setup
cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
sed -e 's/dev\/ad0/dev\/mirror\/gm0/g' </mnt/etc/fstab.orig >/mnt/etc/fstab
echo 'swapoff="YES"' >>/mnt/etc/rc.conf # for 5.3-RELEASE only
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf

# instruct boot stage 2 loader on first disk to boot
# with the boot stage 3 loader from the second disk
# (mainly because BIOS might not allow easy booting from second ATA disk
# or at least requires manual intervention on the console)
echo "1:ad(1,a)/boot/loader" >/boot.config

# reboot system
# (for running system with GEOM mirror on second disk)
shutdown -r now

# make sure the first disk is treated as a really fresh one
# (also not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad0 bs=512 count=79

# switch GEOM mirror to auto-synchronization and add first disk
# (first disk is now immediately synchronized with the second disk content)
gmirror configure -a gm0
gmirror insert gm0 /dev/ad0

# wait for the GEOM mirror synchronization to complete
sh -c 'while [ ".`gmirror list | grep SYNCHRONIZING`" != . ]; do sleep 1; done'

# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on first disk
# as it was synchronized from second disk)
shutdown -r now

GEOM mirror Approach 2: Single Slice, Preferred, More Flexible

The following is a step-by-step command list for remotely converting a production FreeBSD 5.3-STABLE system from a single-disk/single-slice (ad0s1) setup to a two-disk/single-slice (ad0s1 & ad1s1) GEOM mirror (gm0s1) setup without the need for console access or Fixit/LiveFS CDROM:

# make sure the second disk is treated as a really fresh one
# (not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad1 bs=512 count=79

# place a PC MBR onto the second disk
# (with a single FreeBSD slice /dev/mirror/gm0s1 as large as the /dev/ad0s1)
# either automatically if sizes fit...
fdisk -v -B -I /dev/ad1
# ...or manually to make sure the sizes fit:
size=`fdisk ad0 | grep ', size ' | head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
# ...and reduce the size by one block because ad0 and ad0s1 else would
# share the same last sector which could lead to ad0 be recognized as
# the GEOM provider instead of ad0s1. Alternatively, you can keep ad0
# and ad0s1 of the same size and hard-code the ad0s1 GEOM provider by
# adding the -h option to the "gmirror label" command below.
size=`expr $size - 1`
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/ad1

# place a GEOM mirror label onto first slice of second disk
# (actually on the last block of the disk slice)
gmirror label -v -n -b round-robin gm0s1 /dev/ad1s1

# activate GEOM mirror kernel layer
# (makes the /dev/mirror/gm0s1 device available)
gmirror load

# place a BSD disklabel onto /dev/mirror/gm0s1
# (ATTENTION: in FreeBSD 5-STABLE before 14-Jan-2005 the
# /dev/mirror/gm0s1 device has to be specified as just "mirror/gm0s1" or
# the bsdlabel(8) will use the incorrect GEOM name "gm0s1" instead!)
# (NOTICE: figure out what partitions you want with "bsdlabel /dev/ad0" before)
# (NOTICE: start "a" partition at offset 16, "c" partition at offset 0)
bsdlabel -w -B /dev/mirror/gm0s1 # initialize
bsdlabel -e /dev/mirror/gm0s1    # create custom partitions

# manually copy filesystem data from first to to second disk
# (same procedure for partitions "g", etc)
newfs -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
dump -L -0 -f- / | (cd /mnt; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1d
mount /dev/mirror/gm0s1d /mnt/var
dump -L -0 -f- /var | (cd /mnt/var; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1e
mount /dev/mirror/gm0s1e /mnt/usr
dump -L -0 -f- /usr | (cd /mnt/usr; restore -r -v -f-)

# adjust new system configuration for GEOM mirror based setup
cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
sed -e 's/dev\/ad0s1/dev\/mirror\/gm0s1/g' </mnt/etc/fstab.orig >/mnt/etc/fstab
echo 'swapoff="YES"' >>/mnt/etc/rc.conf # for 5.3-RELEASE only
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf

# instruct boot stage 2 loader on first disk to boot
# with the boot stage 3 loader from the second disk
# (mainly because BIOS might not allow easy booting from second ATA disk
# or at least requires manual intervention on the console)
echo "1:ad(1,a)/boot/loader" >/boot.config

# reboot system
# (for running system with GEOM mirror on second disk)
shutdown -r now

# make sure the first disk is treated as a really fresh one
# (also not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad0 bs=512 count=79

# place a new PC MBR onto the first disk
# (with a single FreeBSD slice /dev/ad0s1 _exactly_ as large as the /dev/ad1s1)
size=`fdisk ad1 | grep ', size ' | head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/ad0

# switch GEOM mirror to auto-synchronization and add first disk
# (first disk is now immediately synchronized with the second disk content)
gmirror configure -a gm0s1
gmirror insert gm0s1 /dev/ad0s1

# wait for the GEOM mirror synchronization to complete
sh -c 'while [ ".`gmirror list | grep SYNCHRONIZING`" != . ]; do sleep 1; done'

# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on first disk
# as it was synchronized from second disk)
shutdown -r now