FreeBSD Ports: What To Do When Things Go Wrong

David Bogen <djb@bogen.org>

Any trained monkey can administer a system that is working correctly. Such systems rarely, if ever, need external human, or simian, guidance. The system gets some sort of stimulus or input, makes decisions, takes actions, and most likely emits some sort of product or information. Easy enough. Watch the big wheels go round.

Good system and network administrators earn their salaries when systems are not working correctly. It is when nothing seems to work correctly, debugging data are scarce and managerial or client pressure is the highest that time spent preparing for disaster pays off tenfold.

Using the FreeBSD ports collection is generally an exercise in knowing what command to type and when. Again, trained monkey work. The secret to using the ports collection well is knowing what to do when the output of the ports collection isn't exactly what you expected.

While the general design and implementation of the ports collection helps to minimize the risk of errors, the involvement of humans, magnetic media, long distances, solar radiation, and pure, dumb luck all conspire to assure that mistakes and errors will occasionally be made by both people and processes.

While some ports are revised and updated without causing a ruckus, updating others almost always causes people some pain. XFree86 updates are notoriously hit and miss. For some (most, even), the update is smooth and the software Just Works. For others, the update causes nothing but headaches for any one of a thousand reasons.

Let's examine what happens when an update of a mythical application, let's call it widget-wizard, goes awry. At the start of this example, we have widget-wizard v2.1.2 installed on our system.

Before we upgrade the software, we'll need to update the ports tree on our system.

If we use portupgrade to manage our ports, the process probably looks something like this:

1. Update ports collection via cvsup
2. Update ports index via portsdb -Uu

For those folks who don't use portupgrade to manage their installed ports, the following steps will most likely look familiar:

1. Update ports collection via cvsup
2. Update ports index via ports make index

Having updated our ports collection, we can now update the widget-wizard port. The actual commands to do this will depend on if you use portupgrade or not.

Once the new version (v2.1.4) of widget-wizard is installed, we fire it up and test it out. Some regression testing reveals some serious, yet subtle, bugs were introduced by the the new version of the software and since we can't live with these bugs, we need to go back to v2.1.2.

Now what do we do? Our ports collection has been updated, so we can't exactly roll back the application via the tried and true make install or portupgrade commands. Getting a fix into the port's code, and then into the ports collection could takes days or weeks.

This is where better planning before updating a port could pull our proverbial posteriors out of the fire. There are several ways to first avoid and then handle a failed port upgrade.

The first is to let other people be the guinea pigs. When an upgrade to a port on which you rely is committed to the ports tree, monitor the FreeBSD ports mailing list for a week or so afterward. If others are having problems with the port in question, they will most likely squawk about it in that forum.

Once you've waited a few days for a port's update to be field-tested by others and you make the decision to update your installed port, there are a couple of other strategies you can employ to make the update and possible rollback smoother.

You can always take a snapshot of important ports via tar before you upgrade the tree via cvsup. In our example above, you could have done the following before issuing the cvsup command:

tar zcvpf /var/tmp/widget-wizard-2.1.2.tgz /usr/ports/misc/widget-wizard

Then, once the upgrade was determined to be a failure, rolling back to the old port would be as easy as:

pkg_delete widget-wizard
tar zxvpf /var/tmp/widget-wizard-2.1.2.tgz -C /
cd /usr/ports/misc/widget-wizard && make install clean

Of course, this may wipe out your config files, so you may need to restore those from backup. For a particularly large port (KDE, GNOME, XFree86), following these steps and recompiling the port is unpleasantly drawn out and may extend the time needed to recover from a bad port upgrade unnecessarily.

If you use the suite of portupgrade tools, some of the disaster preparation tools you need are built right in to the upgrade process. Backing up your current installation, including your config files, is quite easy. The following command will create a backup of the currently installed port:

PKG_TMPDIR=/var/tmp/widget portupgrade -b widget-wizard

After the portupgrade command finishes, you will have a file named widget-wizard-2.1.2.tgz in /var/tmp/widget.

Unfortunately, portupgrade always tries to install the latest, greatest version of a port. So, even if you point portupgrade to the newly created package, it will still try to install widget-wizard v2.1.4. As such, you'll have to combine portupgrade with the basic FreeBSD ports tools to install the backed-up port.

To rollback your widget-wizard port, simply use the following sequence of commands:

pkg_deinstall widget-wizard
cd /var/tmp/widget
pkg_add widget-wizard-2.1.2.tgz

If, for whatever reason, you need to roll back the actual port files, you have several avenues open to you.

The surgical strike method involves using the FreeBSD CVSweb implementation to examine what files changed between version of the port. Once you have identified the versions of various files you need to recreate the old version of the port, you can download the files, place them in the proper directory, and re-make the port.

The second method involves less human eyeball time, but suffers from being overly broad at the same time. If you know roughly the last day and time the port with which you are struggling used a version that worked for you, cvsup can revert your ports tree to that time and date.

If you use a single supfile for your ports tree, following this method will cause your entire ports tree to revert to the time and date you choose. As such, you may wish to create a new supfile that will only update the ports collection that contains the port in questions. In the example above, my supfile might normally contain:

*default tag=.
*default host=cvsup11.freebsd.org
*default prefix=/usr
*default base=/usr/local/cvsup
*default release=cvs delete use-rel-suffix
ports-all

Since, I want to revert widget-wizard (which is in the misc collection), I'll write a new supfile to revert just the misc collection. Let's assume for the sake of argument, that the widget-wizard port worked well for us when we used the 13:30 UTC 11 January 2004 version of the port. Given that, our targeted supfile looks like this (changes are preceded by a blank line):

*default tag=.
*default host=cvsup11.freebsd.org
*default prefix=/usr
*default base=/usr/local/cvsup
*default release=cvs delete use-rel-suffix

*default date=2004.01.11.13.30.00
ports-misc

Using the targeted supfile above with cvsup would revert just the misc directory in /usr/ports directory tree. Once cvsup finishes rolling back the ports tree, we could use portupgrade or the normal make install process to install a working version of widget-wizard.

Of course, this means that every other port in /usr/ports/misc is now an older version, so you may find yourself re-updating /usr/ports/misc to a current version once the old version of widget-wizard is successfully installed.

Like those living in areas prone to natural disasters might tell you, it does no good to prepare for an earthquake/tornado/hurricane/flood/etc. after the fact. When you need drinking water, light, and food, supplies laid in before disaster struck are worth their weight in gold.

Updating a critical port is much the same. You need to prepare for failure before it strikes. Test the port on a second system, back up the port's files, back up the port itself, etc. Whatever method you choose to guard against disaster, time spent doing so will repay you ten times over once disaster strikes.