Interview with Jan Schaumann

By Emmanuel Dreyfus <manu@netbsd.org>

NetBSD can also be used as a desktop system, although few people use it that way. This is easily illustrated by the fact that the NetBSD distribution is not desktop-ready: it requires some additional setup to get a nice desktop, while most Linux distributions will simply give you that in a bootable CD-ROM.

Jan Schaumann has been an important contributor to the NetBSD project for several years. He spent a lot of time working on the NetBSD package system, known as pkgsrc, and he currently uses NetBSD as his desktop system. We will try to learn from his experience during this interview.

JS: Before we can answer this question, we first need to actually clarify what makes a system suitable for deployment as a desktop, what exactly the requirements of a such a system are. We need to distinguish between single-user desktop machines and large installations of desktop workstations in a multi-user environment. These large systems pose requirements to the operating system that may not be immediately obvious.

From a user's point of view, the daily experience of working with the desktop should not be any different under Linux or NetBSD. The same applications are available for both platforms, and they behave the same way. The user does not need to know what the underlying operating system is as long as, for example, his mail client, browser, shell and text editor are available. Such an installation is ``user-friendly'', a word that is often misunderstood in this context as administrative requirements are lumped together with the users experience.

The person in charge of maintaining the desktops may have a significantly different experience trying to keep the systems up to date under one operating system than under another. The base operating system needs to be easy to install, maintain and update; this includes the kernel and the userland, all the basic tools that make an OS complete. In addition, there is an obvious need for a mature software package management system.

Now any competent system administrator can probably take the linux distribution du jour and install it as an easy to use single-user desktop system, but I'm willing to argue that NetBSD lends itself much better to the installation and maintenance of a large number of workstations for the following reasons:

It provides a complete OS with a complete userland from the same ``vendor'', obsoleting the need to try to track different development branches for the different tools. It is built from a single source tree (with one central authoritative repository), simplifying system maintenance and upgrades and making it easy to make local modifications and changes if necessary. Finally, NetBSD has an outstanding security track record and a simple to follow release engineering process, allowing operating system upgrades to be planned well in advance.

These are features that make NetBSD ``admin-friendly'', a rather crucial quality for a system that needs to be deployed on a large scale. The pkgsrc framework then adds the needed software package management tools to maintain the number of applications that are most important to the end-users and that make up their experience, the perceived ``user-friendliness'' of the desktop system.

JS: pkgsrc is a source based software package management framework. It is similar to FreeBSD's ports collection (from which pkgsrc was originally derived) or to Gentoo Linux's portage system. pkgsrc has an extensive user guide available from our website at http://www.NetBSD.org/Documentation/pkgsrc/. This documentation includes general user information as well as information on how to create your own, brand-new pkgsrc files (ie packaging a piece of software). It is also available within pkgsrc, of course, and can be found in 'doc/guide'.

Building software via pkgsrc ultimately yields binary packages, which can be deployed on other machines via the pkg-tools that are part of pkgsrc. So you basically get the best of both worlds: you can compile the software yourself, which may take a bit more time but allows you to customize it according to your preferences and requirements, or you can install and update binary packages, packages that you may have built yourself to suit your needs or packages that you have retrieved from the NetBSD mirrors.

The process of building software from source in pkgsrc consists of a number of steps, all of which are automated: The software version is looked up in a vulnerability database (and a warning issued if it is found to be vulnerable), the distribution files are retrieved from the internet, their integrity checked against a digest (to prevent accidental installation of a trojan), any dependencies are checked and if necessary built themselves, any patches applied, the software compiled and installed, all files belonging to this software registered in a database and finally a binary package created.

The framework is largely written in Makefiles and shell scripts, so the way one might install the firefox browser and create a binary package is as simple as:

$ cd /usr/pkgsrc/www/firefox && make install package clean

Now with a complex package like firefox, this might take a little while, depending on your hardware. It might be worth considering doing all package builds on a beefy machine with a fast processor and lots of memory. The NetBSD developers continually run so-called bulk-builds of all packages on fast machines and upload the resulting binary packages to the ftp site.

Finally, I should mention that the pkgsrc team does it's own release engineering and security auditing. With the huge number of currently available software continually being developed, software updates occur constantly. But not every user will necessarily want to upgrade each package to the very latest version; often, users want to keep their stable software running as is. This is why the pkgsrc team branches a stable pkgsrc release every four months. The packages on this stable branch will only be updated if crucial bug-fixes are released or a security vulnerability is fixed.

If a vulnerability is found in a given package, it is added to a specific database. Users can easily match the set of installed packages on their system against this file via the 'audit-packages' command, provided by a pkgsrc package (found, quite appropriately, under 'security/audit-packages' in the pkgsrc source tree). Each vulnerable package found is noted, and the administrator can then choose whether or not to remove the package, upgrade the package to a non-vulnerable version or maybe whether or not the vulnerability does not apply to his or her particular setup.

If a package is found to be vulnerable, the package maintainer is contacted and asked to investigate the vulnerability, to see if a newer version is available or if software patches can be applied that solve the vulnerability. If a fix can be found, then it is also pulled up to the latest stable pkgsrc branch.

These two features, security auditing and stable pkgsrc branches, are invaluable to anybody who maintains a large installation with lots of packages. It is also noteworthy that the pkgsrc maintainers are very careful in when they update a package. Not every piece of software is necessarily updated just because a newer version has come out. Instead, the package maintainer evaluates the new features and tests the package in a variety of setups to make sure that the update does not break anything and generally applies the principle of least surprise for the user. Sure, sometimes this means that yesterdays version of wizbang-widget is not yet in pkgsrc, but if that is the case, there is probably a very good reason.

JS: Well, as I mentioned above, there is a large number of binary packages available from the NetBSD ftp site. You are welcome to simply install your software via the pkg_add(1) command from there:

$ pkg_add ftp://ftp.NetBSD.org/pub/NetBSD/packages/2.0/i386/All/firefox-1.0.7.tgz

Or, if you are too lazy to type this long URL for each package, just set the PKG_PATH environment variable in your shell startup files and then add the package:

$ export PKG_PATH=ftp://ftp.NetBSD.org/pub/NetBSD/packages/2.0/i386/All $ pkg_add firefox

This command will retrieve the firefox binary packages and all the dependencies that are not currently installed from the remote location and install them on the local system.

If you have a very slow (or even no) internet connection, then you will have to retrieve the necessary packages in some other way. This, interestingly, also works for both the source based and the binary packages approach. The pkgsrc guide contains an example of how to use the 'make fetch-list' command to create a list of all the files that need to be retrieved in order to build a package. This list is created as part of a shell-script that you can then copy to a machine with an internet connection, where you execute it to retrieve the files, which you can then burn on a CD, for example, to take back to the machine that does not have a (fast) internet connection after which you can build your software without any problems.

For binary packages, you may wish to download all the packages from our ftp site and burn them to a CD. With such a CD, you can use pkg_add(1) just like before:

$ export PKG_PATH=/mnt/cdrom/i386/All $ pkg_add firefox

ED: My experience with pkgsrc is that initial installation is easy but upgrades can be tricky. This is no surprise, since there are many software components interacting together. How do you perform a package upgrade?

JS: Yes, software upgrades are often difficult, especially if a package influences a large number of other packages. The package dependency graph for any but the most trivial setup is surprisingly complex. (If you wish to visualize how your packages are connected to each other, I recommend the 'pkgtools/pkgdepgraph' package, which generates a dot language description of the installed packages. This output can be fed to the 'graphics/graphviz' package and can then be used in surprisingly many ways.)

[See a dependency graph: http://www.cs.stevens.edu/~jschauma/netbsd/lapdog.png]

The pkgsrc system offers a number of ways to perform system upgrades. Fortunately, it will automatically sort out for you exactly which packages need to be updated after one package is upgraded. If you are using the source based approach, this can be as easy as:

$ cd /usr/pkgsrc// $ make update

This easy to use procedure does come with a caveat, however: pkgsrc will determine all packages currently installed that require this package, uninstall them, build the new version of this package and then rebuild all the packages that were previously uninstalled.

This is necessary since an update of, say, an important library does require that all other applications that use this library are linked against it. Yes, yes, in theory this would not be necessary if the library has not changed its ABI; in practice, however, this does not always work and by rebuilding the dependencies, we ensure that they are actually working with the new version.

As you probably have noticed, this entails that at some point your system is without the packages that were installed originally. But you're not performing this step on your production system anyway, now are you?

If you are absolutely certain that your installed applications will work with the new version of the package which you are updating, then you can -- at your own risk! -- use the 'make replace' target, which will indeed switch the old version against the new version without rebuilding the dependencies (it does, however, note the new dependency of the other packages on the newer version). This approach must be used with caution! If I perform this step, I usually run a quick check against all installed binaries to check that they do find all required shared libraries.

As I said, upgrading packages is not something that you want to be doing on your production systems (ie the actual users' workstations). Instead, I recommend having a machine set aside that builds the software and then upgrade on the target system via binary packages. There are several ways of doing this, and depending on your resources you may wish to choose one over the other.

The easiest way is of course by simply downloading the newer binary package from the NetBSD ftp mirror. Note, however, that the NetBSD pkgsrc team can only provide binary packages for the latest stable branch, so if you're following the current development branch, binary packages for the new version may not yet be available, meaning you have to build them yourself.

If you have the necessary resources, and are maintaining systems with a large number of packages, it might be a good idea for you to run your own bulk-builds, possibly continually. This ensures that you always have the latest binary packages ready for installation. The pkgsrc guide, which can be found in pkgsrc/doc/ in addition to the URL I mentioned above, provides instructions on how to set up such a bulk-build.

If you do not wish to build all of the >5600 packages currently in pkgsrc, then you can also use the bulk-build approach to only build a subset of available packages, based on your specification.

Another way to do this is by way of the 'pkgtools/pkg_comp' package. This wonderful little tool will set up a chroot for you in which to build a pre-defined set of packages. You simply add the packages you wish to have ready to this file, update your pkgsrc tree and run the program and it will go right ahead and create the binary packages for you. In addition, it lets you build packages for different versions of NetBSD; so if you are using, say, NetBSD 2.0 on one set of machines and 3.0 on another set of machines, you can build binary packages for both versions in separate chroots on a 3.0 machine.

JS: Well, automating the maintenance of a number of desktop systems is not fundamentally different from automating the maintenance of a number of server systems, and every system administrator has his or her choice of configuration management tools.

At Stevens Institute of Technology, where I work as a system administrator, I maintain about 85 NetBSD workstations on faculty desktops and public laboratories. The system I use to deploy updates is described in detail in my paper ``NetBSD/Desktop -- Scalable Workstation Solutions'', which I presented at EuroBSDCon 2004. In one sentence, I build software on a build-host inside a chroot, install it into a different chroot which contains the workstation image and rsync updates to all clients.

It is worth noting, I think, that on these workstations have well over 1100 packages installed, among them most common window managers and desktop environments (including WindowMaker, blackbox, KDE and GNOME), various browsers (including Netscape, Firefox, Mozilla and Opera), numerous software development tools (ranging from autoconf and automake to full-featured IDEs like eclipse), programming languages (such as Sun's JDKs, Python, Perl, Ruby and Scheme) and office suites and applications (including OpenOffice, gnumeric, gnucash, abiword and KOffice). In other words, they contain a complex and full desktop environment for a multitude of different users.

As you can imagine the dependency graph of this installation is not what you would call trivial, and updates to any but the leaf-packages must be done very carefully. But the pkgsrc system does indeed help me remain sane. Sure, it's not without flaws, and often I curse it and all of the packages we use, but in the end, it simply ``sucks less'' (tipping my head to my favorite mail client).

The list of software that we have shows that a NetBSD desktop does really not look very different from a Linux desktop. While most of my users don't even notice what kind of Unix flavor the operating system they're using is, I do get asked every once in a while why we do not run Linux. My answer usually remains the same: why should I?

JS: Actually, I hardly spend any time at all on this. Again, as I mentioned above, the software is exactly the same software that is available for Linux, so if that software comes configured for general purpose usage on any unix-like desktop, it won't be any different when installed on NetBSD. For example, when I install, say, KDE or GNOME under NetBSD via pkgsrc, then all I have to do is tell my users to enter the command 'startkde' or 'gnome-session' in their ~/.xsession file, and a completely usable desktop environment will start without the system administrator having to tweak any files.

However, there are two features that pkgsrc provides that are very important to me: For one thing, pkgsrc makes it trivial to customize installed packages as long as the original software allows for central configuration and it ensures that I will not lose the configuration during the next upgrade.

Most packages that do allow configuration on a system wide level provide an example configuration file that is installed under /usr/pkg/share/examples//. You can copy that file to /usr/pkg/etc/ and modify it as needed. If you upgrade the package, the file that you placed in /usr/pkg/etc will only be removed if it does not differ from the provided example file. That is, if you make changes that are local to your site in such a file and then upgrade the package, you won't accidentally lose your configuration file!

Secondly, as you know, there often are software packages that make general assumptions about where certain files can be found, what kind of feature are to be enabled and where to write certain data to. It has been my experience, that under the various Linux distributions this is a very brittle setup (or a bold assumption, whichever way you may want to look at it) and often one ends up with one package writing to files in /opt/foo/ and another to /var/foo, for example. The software, when packaged and prepared for pkgsrc, however, is usually configured in such a way that it will always look in specific default locations but it will allow you at build time to specify other locations via certain configuration parameters. Similarly, packages are kept flexible and allow you to turn on (or off) specific features via so-called PKG_OPTIONS.

These two features allow me to actually go ahead and install software from pkgsrc without having to think much about it and it will work out of the box without any site-specific configuration.

JS: Nope. Given the number of applications I have installed, I really can't be bothered with trying to fiddle with each and every one of them. After it's been installed, it's up to the user to configure it to their liking. The applications menus from KDE are populated automatically with all KDE and most GNOME apps, and for GNOME it includes all GNOME apps and some, but not many other applications.

WindowMaker, blackbox, xfce, 9wm, fwvm, larswm, twm and the other window managers I have installed all need to be configured by the user, that's true. But I really don't see how that's different on any Linux distribution: I've seen Linux desktops where the applications menus contained items that simply weren't installed or were installed in different locations.

JS: Well, while I've been praising NetBSD and pkgsrc, it remains a fundamental truth that any system is only as good as the system administrator who maintains it. I'm not saying that it's not possible to set up another operating system with another software package management system in a consistent and scalable way, but I hope I could show that contrary to some misconceptions NetBSD does indeed lend itself quite well for use as a desktop system, not in spite, but particularly because of its ``admin-friendly'' features.

Thanks for the interview, Emmanuel!