Daemon's Advocate

By Scott Long and Robert Watson

First of all, the last decade has been ripe with cooperation between all three of the major BSD projects. Each project gives and takes from the others, and there are a number of developers that have commit privileges to multiple BSD projects. Drivers, infrastructure, bug fixes, and features readily flow between projects. This benefits everyone, especially since it allows each group to focus on unique aspects of the system without having to be bogged down with other aspects. As the old saying goes, FreeBSD is about performance, NetBSD is about platform portability, and OpenBSD is about security.

So is that still the case? The NetBSD advocates are quick to claim that NetBSD 2.0 now beats FreeBSD in both performance and features. Fortunately, that is just not true. There is a very long list of reasons why FreeBSD is an excellent operating system and an ideal choice for the enterprise and the desktop. Briefly:

• Netgraph provides unparalleled flexibility to build complex network environments. Netgraph modules are available for packet filtering, tunneling, redirection, inspection, and injection at any point in the network stack in a transparent and quick fashion. Modules can be stacked together like bricks to meet just about any need. Developing custom modules is also easy and very well documented. There simply is not anything else in any other OS that is as flexible, easy to use, and full-featured as netgraph.
• GEOM provides to the storage stack what netgraph provides to the network stack. Transformations like mirroring, striping, spanning, and encryption can be configured for any storage object from the filesystem on up. The Vinum volume manager was recently converted to use GEOM and now provides high-availability and high-reliability redundancy to any storage object. While NetBSD recently imported Vinum, it took the older, less stable and less functional version that has since been deprecated by its author in FreeBSD in favor of GEOM-Vinum.
• Advanced network features and protocols such as SACK, NFSv4, SYN-cache/SYN-cookies, compressed TIME_WAIT, and accept filters allow for fast, secure, and scalable network operations in an ever-increasing hostile and busy Internet. Packet filters like IPFW and PF provide advanced filtering, shaping, and NAT sharing. FreeBSD continues to run some of the busiest and most important network sites in the world with these technologies.
• Outstanding desktop and laptop support is provided by a number of technologies. Nvidia develops and distributes native 3D drivers for its graphics cards for FreeBSD. A team of FreeBSD developers works closely with engineers at Intel to provide the best ACPI power management support available in an open source operating system. The Gnome and KDE desktop environments work flawlessly under FreeBSD thanks to another team of volunteers who work closely with those projects.
• The "Ports" collection provides one-step support for over 11,000 3rd party applications. Compile-time and run-time dependencies between applications and libraries are tracked and handled automatically, eliminating conflicts and incompatibilities. Pre-compiled binaries are available for nearly every supported package for quick and easy installation. This system continues to be one of the crown jewels of FreeBSD and has been copied by other OSes due to its overwhelming popularity.
• Many commercial vendors also support FreeBSD. Companies like Intel, AMD, LSI, Adaptec, and 3Ware, just to name a few, provide development staffing, direct developer resources, and end-user support for many of their products. The result is high-quality drivers, applications, and platform support for a wide range of modern hardware.
• Continuous testing and QA is performed by a number of teams within the FreeBSD community. Tests are runs every day that range from simple full-tree compile runs to intensive network, I/O, and computational stress tests. Developers receive status emails and bug reports to help identify, track, and resolve defects. While no amount of testing is perfect and bugs do slip through, the testing that exists vastly exceeds the efforts of most other open source projects and contributes towards every FreeBSD release's high quality.

NetBSD 2.0 is a significant step forward for NetBSD, but the large amount of stagnation cannot be overlooked. Their claim at high portability should have been leveraged years ago to make them the leader in embedded OSes. It's great that NetBSD is committed to supporting legacy architectures, but how does the effort to do so benefit modern architectures or encourage wider use and more adoption of NetBSD?

And while NetBSD now supports SMP, it uses the same low-efficiency model that FreeBSD used previously. Scalability is significantly limited because only one CPU at a time can access kernel services or drive hardware devices. The whole point of the 'SMPng' project for FreeBSD 5.x is to eliminate this problem and provide fine-grained parallelism in the kernel. Converting the traditional BSD design to this model is not trivial, but the work on this is very much alive, and each FreeBSD 5.x release is faster, more scalable, and more stable than the previous release.

All of the open source BSD's have a place, whether it's OpenBSD, NetBSD, or FreeBSD. Each continues to excel at what it has shown to be good at, and I expect the sharing and goodwill between them will continue. And in that vein, FreeBSD is still the 'silent workhorse' that runs corporate networks and powers advanced appliances. However, it's time to drop the 'silent' part and start loudly advocating it. FreeBSD is an outstanding OS, and developers and users should be proud of that fact.

-- Scott

Robert Watson Responds to Scott

I agree that the "NetBSD is more scalable than FreeBSD" conclusion stemming from a previously posted set of benchmarks was bogus: scalability should be measured as a property of real-world applications, not the constant in the file descriptor allocation routine. I.e., if you really want to measure scalability, run a web benchmark or a database benchmark. If NetBSD still comes out faster in a properly run macro-benchmark, then that's an important thing to know, and an important target -- micro-benchmarks are nice to optimize for, but the big picture is more important, and not mentioned in the benchmark paper. Everything was fine in the paper, with the exception of the abstract/conclusion, which had one of those "then a miracle occurs" moments in reasoning. :-) So I wouldn't dismiss the technical results, but the "headline" was pretty out of line with regard to the technical results -- this follows in an increasingly long history of weird open source "scalability" benchmarks, in which a set of micro benchmarks is used to justify an overall argument that isn't supported by the technical content. By reaching less far, the argument would become more accurate. As it turns out, it's a lot easier to run micro-benchmarks properly than macro-benchmarks properly, which is probably why micro-benchmarks predominate.

I think it is the case that lately the NetBSD folk have been doing a more effective PR job than they have been previously, and have managed to grab some headlines as a result. Our response, in as much is one is needed, should be three-fold: where NetBSD can demonstrate a quantitative performance of feature benefit over FreeBSD, we should figure out why, and fix it where it makes sense and is important. We should do some comprehensive benchmarking of our own, and use them to improve our system as well as document its benefits. The NetBSD benchmark was a classic example of micro-benchmarks feeding performance optimization, and vice-versa -- not without benefit, but true nonetheless. However, our primary objections to it haven't been that we sometimes may not win every micro-benchmark, but that it missed what's really important in scalability -- scalable application performance on the platform.

There are some important ways that out SMPng/KSE architecture should (and does) perform really well; for example, our threading library runs threads on more than one CPU at a time by default (which the NetBSD SA implementation currently doesn't do), and we can execute parallel system calls in-kernel, which should be quite measurable (and I've seen results suggesting that it really is). However, because we took such a cut in performance early in 5.x development, we've not been doing continuous or effective long-term benchmarking. I have a nice history of MySQL results improving over the last year -- basically doubling (or more than doubling) in performance as we improved, especially on SMP -- but that's hardly comprehensive. We've built excellent project infrastructure for build management, packaging, etc., but we need to work more effectively as a project, not just as individuals, to manage performance the same way.

We should do a better PR job of identifying areas of particular strength -- your list of features would make the great beginnings of a white paper on FreeBSD network stack technology, for example. Many of the features there have set the standard for implementation in other systems -- kqueue is now widely used across many platforms, and those that don't have kqueue try to imitate with a bit of NIH on the side :-).

Finally, we should continue our heavy investment of development and other resources into the FreeBSD project -- we're a very large project with a very large number of extremely competent developers. The BSD platforms have defined operating system technology for many years, and in many ways continue to do so. We should make sure we continue to do that. Projects like SMPng, KSE, etc., have come an incredibly long way in the past few years, and reflect both the huge challenges they involve, and the level of investment we've made. Commercial UNIX vendors such as Sun easily spent far more in the way of resources to bring these features to Solaris, and probably took even longer than we did to bring them to fruition. We have the benefits of a very open development process, and we take advantage of them. While there's more work to do, we can actually see the finish line on these projects. With recent work by Jeff on VFS/UFS, we're reaching the point where a significant percentage of our kernel is fully threaded, parallelized, and able to run preemptively. The benefits are starting to pay off, so now is the time we can reap the best benefits from our careful work. We should now be able to perform optimization and illustrate some of the potential gains from the more mature kernel architecture.

So I guess my message is this: we should focus our energy in demonstrating (and making sure) FreeBSD is the best platform out there. We've done an incredible job, but we need to keep doing it. This includes doing a better job with PR. We need to do exactly what you've laid the groundwork for: making an argument for FreeBSD as a platform.

-- Robert N M Watson