Ike Notes in BSDCan 2005

By Isaac Levy

I understand more notes are going up on the BSDCan website: http://www.bsdcan.org/

With that, after a super-busy week returning home, this article is a collection of e-mail sent to the NYCBUG mailing list as 'ike-notes' on the conference, (a bit heavy on the FreeBSD end of things, as a lot of my day-to-day relies heavily on FreeBSD...)

Kernel Debugging with Greg Lehey: http://www.bsdcan.org/2005/activity.php?id=68

I had the pleasure of attending Greg Lehey's 'Kernel Debugging Workshop', which happened on the Wednesday preceding the regular conference schedule- and it was great. I live in user-space for most of my development work and hacking life, so it was a real eye-opener to dive into the Kernel for an entire day... a lot of heavy information to take in all at once :) One of the most striking things about this lecture was that Greg uses Firewire for debugging, insomuch as firewire PCI cards are cheaper than serial, and MUCH faster. Firewire gives direct access to system memory, so it's great for debugging. Mr. Lehey's 'most correct' lecture notes are available online, and are currently the most definitive documentation available on debugging BSD kernels:

http://www.lemis.com/grog/Papers/Debug-tutorial/tutorial.pdf 1.2mb PDF

Robert Watson gave two great presentations, one on SMPng, the FreeBSD Network Stack, where he discussed the accomplishments and current challenges for improving SMP on FreeBSD at a low level. Watson, and the folks working on SMP, REALLY have their work cut out for them here- and their general direction is really solid. For me, it was cool to see dev. details for things I rarely think about- because they just work :)

His second lecture, "TrustedBSD Audit: BSM Security Event Logging for FreeBSD", was REALLY eye-opening. Basically, this work revolves around creating hooks in the kernel which allow for total event logging for system activities. Every time a file is touched, a process started, etc... Two historical notes struck me, first being this was implemented long ago in SunOS, according to US military specifications. Second, that Apple hired McAffe Research, (where Robert Watson works), to implement this work in Darwin 8, (OSX Tiger), for use with Spotlight! (was anyone but me wondering how this worked?). Apple was convinced to release the code under a BSD (*not* APSL) license, and this TrustedBSD project code is to be merged into FreeBSD 6.0. Now THAT's cool, and a great example of how Apple is contributing back to the Open Source community!!! (Too bad apple marketing doesn't talk about low-level open source dev :) More info: http://www.trustedbsd.org/

Poul-Henning Kamp was everywhere it seemed, he gave a great presentation titled 'ioctl is just soooo 1980ies', (slides included a picture of the A-Team, among other important 80's ideas/ideals). With that, he went through a history of the ioctl system call, it's importance, it's kludginess, and his proposed solutions- which are closely tied to his other abstraction-type works in GEOM. Coming soon to a FreeBSD Kernel near you, g_ctl and nmount. Basically, to summarize a fairly complicated topic, PHK is trying to abstract every device, and provide a unified interface through the kernel to devices. Cool stuff- can't wait until the dust starts to settle on this stuff... :)

The lecture slides are here: http://phk.freebsd.dk/pubs/bsdcan2005_ioctl.pdf

- Personal side-note, PHK discussed passing text into the kernel, or even structured text in the form of XML, which piqued the counter-argument from our own Bob Ippolito- (as I sat in the back grinning... :)- Bob and I had been up into the wee hours the night before, discussing the text/xml vs. binary formatting issue, sparked by changes to Darwin/OSX metadata files going binary. Bob argued that the binary formats, if accompanied by proper tools to construct the binary data, can have much saner real-world application for critical data i/o operations in a given program. I tend to agree with Bob on this one, citing a recent adventure in hell, implementing the DMOZ xml dump in a client project- in a nutshell, XML breaks, text parsing breaks, and it's extremely difficult to focus on what one is *doing* with the data- whereas things like database dumps and the like at the least help focus a developer on the actual data, by taking care of the formatting and presenting focused apis/applications for using the data.

6 of one, half-dozen of the other- this entire argument is not a new one. http://www.catb.org/~esr/writings/taoup/html/ch05s01.html

Regardless of the semantics of how it all works out, I'm really exited to see PHK's replacements for ioctl take off like GEOM has!

Really cool stuff, Ryan McBride gave a lecture on network stack randomization in OpenBSD- which I thought was really cool stuff. Basically, as always, the OpenBSD crew makes software that belongs in the MOMA, IMHO- pretty intense stuff. Ryan discussed the various sources of entropy that get xor'ed into a pool of packed ids for the tcp/ip packet stack. The talk was brief, which led to a great Q&A with the bulk of the OpenBSD core team in attendance, which went into other randomization in OpenBSD, (pid randomization, ProPolice use for memory pointer stack randomization, etc...), and quickly spun into some general hardcore OpenBSD Q&A quality time.

I'm sad to say I missed Bob Beck's lecture on Spamd, as well as Henning Brauer speaking on OpenBGPD, but am pleased to say there was ton of great conversation at the bars about various topics with the OpenBSD crew- which made up for it.

While having drinks, Henning Brauer explained briefly how I could replicate the functionality I love from FreeBSD's jail facility (which I was speaking about at the conference), using OpenBSD. Basically, it involves chroot'ing all the OpenBSD userland apps, and using PF to restrict an IP alias interface to the user process which is running the chroot. (PF now can filter packets by user process). I am currently hacking around with this procedure at home- mostly getting to know more about PF and hacking around... (now that PF is native in FreeBSD, crossover will be MUCH easier.)

In another discussion, Mathieu Sauve-Frankel (Matt) explained in greater depth some of the reasoning behind why there's little interest for a jail facility in OpenBSD- basically that their concerns are with more fundamental security ideas, and that jailing bad software, is still jailing bad software... an attitude I can totally agree with. However, I conversely argued basically that *all* software is bad software, and there are other applications for a jail facility... Beer and food was served, and conversations switched gears a million times.

We called our own Mikey in NYC, who sadly could not attend, but will be in Canada soon for the OpenBSD Hack-a-thon!

Discussions I had with all the OpenBSD folks were really fun, since I use (and love) so much in FreeBSD, we were coming at the same problems from opposite ends of the universe. Needless to say, after the Conference, I'm now making my duct-tape-computers lab a *much* more heterogeneous environment all around...

Exiting Tech, Exiting People: -- Several topics which have been swarming around NYCBUG seem to be moving foreword,

- CARP/PF native and stable on FreeBSD 5.4! This is SERIOUSLY exciting. I had the pleasure of drinking with Max Laier, who is the FreeBSD commiter for PF and CARP. I'm told Gleb Smirnoff and Pyun YongHyeon started the port, (last year give or take).

- OpenBSD 3.7 was released a bit early on CD at the conference, but it's out and available this week to the world.

- The Intel hyperthreading vulnerability issue was disclosed and reviewed in detail, but we've all seen this by now, check here for more info: http://kerneltrap.org/node/5103

- FreeBSD Roadmap Crib Notes- After chatting with Michael Lucas, ( http://www.oreillynet.com/pub/au/67 ) and picking the brains of a few FreeBSD committers who were there early for the FreeBSD dev. summit, I got the basic skinny on the 5.x, 6.x, and 7.x release schedule for FreeBSD. In a nutshell, things look quite good all around- and I'm in the testing process to *finally* migrate all my production life to 5.4. FreeBSD 6.x is way less feature-packed than 5.x, with less fundamental changes- the aim is stability and speed. Michael Lucas noted that the 4.x FreeBSD's were soooo good, it's simply a hard act to follow. With that the FreeBSD committers are all working hard to exceed the speed and stability of the 4.x branch. FreeBSD 7.x was mentioned as well, as the committers have decided to move a bit faster with release versions- fewer features and functionality packed into forthcoming releases, (fewer than what we've got with 5.x!).

With that, things look great for the future of FreeBSD with regard to stability and speed.

- I got to briefly see Dru Lavigne again ( http://www.oreillynet.com/cs/catalog/view/au/73 ), which is always a pleasure- but didn't get much time to talk. She's been mad busy with the BSDCert crew, (Jim Brown, Mark Spitzer, and George Rosamond from our NYC crew are heavily involved too...) and and they held a great BOF session to address the community needs for a BSD certification program. If you haven't done so already, go fill out the survey so the certification is relevant to YOU when it comes out!!!

http://www.bsdcertification.org/downloads/BSDCertSurvey01_en-us_ann.html - With all of this, there were soooo many great people, conversations, ideas, and so much hacking around- I hope more of us can make it to BSDCan next year. This conference is totally a sacred holiday to me now :)

############## Special thanks should go out to Dan Languille, who PUTS ON BSDCAN ALL BY HIMSELF. He's just plain awesome, period. ##############