The Daemon's Advocate

by Greg Lehey <grog@lemis.com>

Anarchies, monarchies and dictatorships

Over the last couple of years we've discussed in some detail how BSD does things, why we do it that way, and what makes BSD projects different from Linux. One of the interesting things is the way the project is run. In the Linux project, Linus Torvalds is at the top, and there are a number of important people who help him. They're the only people who can directly change the kernel sources, but there's no clear structure (well, I can't see one, anyway). In many ways it's reminiscent of a feudal monarchy.

Contrast the BSD approach: NetBSD and FreeBSD have a core team (the FreeBSD term) or core group (the NetBSD term), a group of people who direct the project. In practice, however, it's not the core team's responsibility to commit changes to the source trees: there are many more committers than core team members.

Why a core team?

So why a core team? How did it happen? Why is there a difference between core team and committers?

I can only really say for sure about the FreeBSD project, though I suspect things are not much different in NetBSD. The projects have been going for quite a while now, and even the ``newcomer'' OpenBSD has been around for nearly 5 years. In that time, things have changed enough that it's worth looking at it from a historical perspective.

The OpenBSD way OpenBSD started later, so it had the benefit of seeing the weaknesses of the core team approach. To quote Theo de Raadt, referring to the current situation: Different people, or groups of people, own different parts/subsystems of the tree based on previous work there or based on wide-ranging testing experience there. As a benevolent dictator, I ensure that people follow that chain of control, and that people get a baseball bat to their skulls if they fail to work together nicely. I also work very hard to make sure that people are aware of who is working in which areas, and act as a hub who connects developers with questions to developers who can help them find the answers. Beyond that, we have further out developers, who sometimes use another more central developer as a hub, though that varies so much, that we refuse the call OpenBSD a core-run operating system. NetBSD policies caused OpenBSD to come into creation, and we strive to not repeat those things. Things are pretty much in the open, and things can get slightly or very out of hand if they need to, in the end, I can always put on the benevolent dictator cap, and we can move on. Above all, I believe that very flexible, certainly divergent, and possibly even inconsistent policies are required to herd more than 10 cats.

In the beginning

In 1992, when Bill Jolitz released 386BSD, the idea of free software was not taken very seriously. The term ``Open Source'' hadn't been invented, and the only serious contender was the GNU project. Bill hasn't kept many friends, but the software community as a whole owes him a debt of gratitude: at a time when free software was considered dubious idea, it was certainly really radical to take a recognized version of UNIX and make it free, at a time when AT&T were charging $50,000 for a source license.

Bill's problem was that he wasn't prepared to cooperate. It was his operating system, and he was going to do it. Of course there were bugs; the initial versions were so buggy that nobody would dream of using them for real work. Nobody expected anything else, however, and so plenty of people set to work to fix the bugs and send the fixes back to Bill. Bill ignored them.

People kept trying; the first people to get fed up brought out their own version and called it NetBSD. Others just collected the patches and kept trying to get Bill to feed them back into 386BSD. Finally the maintainers of the patch kit gave up, took their patched version of 386BSD and called it FreeBSD.

By this time, I'd estimate that there were around 10 people in each project, mainly people who knew each other. That wasn't enough, of course: many more people were needed. But how to ensure they didn't take over? The people who were there first made the decisions. The core team was born.

Changing times--the FreeBSD viewpoint

This was over five years ago. Open Source software was mainly software for hackers' personal use, not something you'd use for Real Work. At the time, round mid-1994, I was using BSD/386 (the precursor of BSD/OS), and I only used FreeBSD for non-critical cases where people weren't prepared to pay BSDI's price. From the release notes for FreeBSD version 1.0, in November 1993, we note that the FreeBSD ``core'' group [sic] consisted of 11 people, three of whom are still in the core team. The document also identifies 18 further ``additional FreeBSD helpers and beta testers'', of which eight are still with us today. You can get a bit of the flavour of how times have changed from the following acknowledgement:

Dermot McDonnell for his donation of a Toshiba XM3401B CDROM drive.

Over the years, the flavour of the project changed. Somewhat to our surprise, FreeBSD became a system which could contend with the best of them. At the same time, more money became available, and now there are a large number of people using expensive hardware to develop the system. At the same time, the core team expanded. To become a member of the core team, you had to be invited by the existing core team. On one hand, this made a lot of sense: if the members of the core team can't get on with each other, not much good is going to come of it. On the other hand, the number of contributors has increased out of proportion: the core team has increased in size from 11 to 15, while the number of other contributors (now committers) has increased from 18 to over 200 (including the core team, however). Clearly the relationship between the two groups has had to change as well.

The present

As Theo observes, it's quite an achievement to keep the peace amongst such a large group of people, most of whom have never met each other. From time to time, differences of opinion arise, and we've seen a couple of ugly arguments in the past. On the whole, the core team did a very good job of mediating, but in the course of time people had problems with the core team:

• The core team appeared to be not doing enough. Problems reported to the core team seemed to disappear into a black hole.

• The core team appeared to be doing too much, meddling in affairs which didn't concern them.

• Some people accused the core team of cronyism, that they were applying two sets of rules, depending on the people involved.

• Some people also pointed to individual members of core who hadn't been heard of for a surprisingly long period of time. This ``dead wood'', they said, was a problem.

The obvious first question is, of course: ``What are the duties of the core team?''. Well, guess what. They were never defined. So it's not possible to say objectively whether the core team is doing a good job or not. Taking a step back, though, it became increasingly clear that there were a number of problems, and the majority of committers thought that the core team should be doing something about it. Since April 2000, there was a lively discussion in the FreeBSD-committers mailing list. This is the mailing list to which all committers, and nobody else, belong.

In this time, over 1000 messages were sent discussing the matters. It's difficult to give an exact summary, since a lot depends on individual points of view, but briefly:

• In April, Jordan Hubbard summarized the current situation:

  I'll furthermore go way out on a limb here and express my honest opinion that core's time actually ``passed'' some months back.

• A couple of people suggested various ways to reform or change the core team, including the possibility of disbanding core altogether and becoming an anarchy, or voting for the core team. A number of people came up with remarkably complicated voting models.

• Finally, Jordan came up with a suggestion and asked -committers to vote on a number of alternatives. Out of 94 votes cast, the most popular were:

  • The idea of core is fine, its membership simply needs a shake-up and some mechanism added for voting in new blood.

    58 votes

  • The idea of core is fine, but some of members simply need to leave.

    12 votes

  • Core needs to be broken up into an oversight/human resources group, leaving architectural decisions to developers.

    9 votes

  • Don't change anything, core is fine the way it is.

    7 votes

  • Disband core entirely and let committers create a new structure in its place.

    7 votes

  Clearly the majority was for a democratically elected core team. More discussions ensued. Some people were concerned that politics would take over from reason, and the people who would get elected would be those who could drum up enough followers, not those who could do the best jobs. A team of volunteers, consisting of Jonathan Lemon, Warner Losh and Wes Peters, got together and thrashed out the existing voting models and came up with the following proposal, on which we also voted:

  • Active committers have made a commit to the tree in the last 12 months.

  • Core consists of 9 elected active committers.

  • Core elections are held every 2 years, first time September 2000.

  • Core members and committers may be ejected by a 2/3 vote of core.

  • If the size of core falls below 7, an early election is held.

  • A petition of 1/3 of active committers can trigger an early election.

  • All elections will be run as follows: Core appoints and announces someone to run the election. 1 week to tally active committers wishing to run for core. 4 weeks for the actual vote 1 week to tally and post the results. Each active committer may vote once in support of up to nine nominees. New core team becomes effective 1 week after the results are posted. Voting ties decided by unambiguously elected new core members.

  • These rules can be changed by a 2/3 majority of committers if at least 50% of active committers cast their vote.

  These ``bylaws'' passed by 117 yes votes to 5 no votes, thus also disproving the concern that committers wouldn't be interested enough to vote for the core team.

  A couple of these provisions look a little unusual:

  • The rationale behind the surprisingly long election period was that, since this is a volunteer project, many people might miss a shorter election period, especially Europeans on multi-week vacations.

  • We spent a lot of time discussing how to vote. We were concerned that if each voter had only one vote, the majority would vote for the same two or three candidates, effectively leaving the remainder to chance. Even worse, it could lead to less than 9 candidates being voted for at all. Initially, we also discussed a ``veto'' vote: ``Don't let <that guy> onto core''. You'll note from Jordan's poll that the second most popular model was ``expel <that guy> from core''.

  The election

  Nominations for candidacy were accepted between 1800 UTC on 5 September 2000 and 1800 UTC on 12 September 2000, after which the election started immediately. It finished at 1800 UTC on 10 October 2000. The results were announced at 1800 UTC on 12 October 2000, just in time for the beginning of this year's BSDCon.

  A total of 17 candidates registered, surprisingly close to the size of the current core team. Only 8 of the current core team stood for election. Campaigning was almost non-existent.

  The morning after

  This article was originally intended for publication at the beginning of the month, but one of the concerns we had about the election was that it should involve as little politics as possible. In order to avoid any suspicion of manipulation, we decided to put off publication until after the election results.

  We now have a new core team. Five of the members were carried over from the old core team, and two of the old core team were not elected. The composition of the team has changed in other ways: five members of the old core team had a non-English native language, but only one member of the new team does. Seven members of the old core team were outside the USA, only two of the new team are. Is this going to change things?

  Let's consider another comparison: others have asked ``how many women are on the core team?''. The answer, both before and after, is ``none''. This isn't discrimination: we currently have no female committers, so it's just not possible. For whatever reason, hacking remains a very predominantly male business.

  So, back to the composition. Will it change anything? It's too early to know. We're expecting more of the new core team, of course. As in the past 7 years, the time ahead will be a learning experience. Let's enjoy it.