I read a great article (subscribers only) in the latest issue of Linux Journal called “Constructing Red Hat Enterprise Linux 4.” Its precursor for RHEL3 is freely available though. If you’re involved in the distro world but don’t subscribe, stop by your local bookstore and pick up a copy.
The articles are by Tim Burke, RH’s “director of kernel development” (sounds like the kernel lead), so they’re biased toward the kernel. But the principles are mostly already abstract or easily applicable to other areas.
I’m going to briefly review some important points from them and point out a few things I found interesting about how RH works. I’ll start w/ the RHEL3 article, since everyone can get at that.
If you want a one-sentence summary, this is it: Red Hat has learned that being in the open-source business works better with an open-source development philosophy.
RH has all these corporate partners. In the V3 story, we see RH gets tons of feature requests from them. Later on in V4, we see partners actually sending employees to work in person at RH. Back to V3: the first encouraging point is seeing refusal to add proprietary additions and hooks, which we’ve seen roughly paralleled in the Linux kernel with examples such as the Philips webcam driver. They requested top-10 lists from their partners, to keep a flood of demands from deluging the “important” ones.
Once he starts getting into the kernel, you’ll notice one of the things RH’s 2.4 kernel was famous for — being halfway to 2.6 without the version number reflecting it. In the V4 story, we see that RH is pushing much harder to get its patches upstream and to keep its maintenance burden at a minimum by using something closer to vanilla.
We see many of these points re-emphasized in the V4 story, such as upstreaming and not allowing feature creep. An example:
There was one incident when we were two hours from shipping the release and a delivery arrived on the loading dock. It was a new computer platform we needed in order to be able to develop and test support for it. The partner was incredulous that we were unable to accommodate.
It also shows how even collaboration within RH is subject to the usual (in open source) time-zone disagreements. They “rarely have team meetings” because of this.
One thing it focused on that Gentoo largely lacks is widespread testing and QA. Ours is mostly done by our community, whereas they do it before handing out the updates. They’ve got tests ranging from compliance to LSB/POSIX to stress tests such as lmbench and bonnie, all of which they run nightly. They only call in real people well after they’ve run the automated tests.
Moving on to the V4 story … I wasn’t impressed with how parts of it were phrased. For example, the beginning reads like an advertisement for RHEL. Also, RH as a whole does a great job of demeaning Fedora users by calling them guinea pigs and Fedora nothing but feature testing for potential inclusion in RHEL.
There are more positive ways to refer to Fedora and its users, folks. =) It’s a community, and you have to treat it as a community of equals, not one of lab rats, because there’s no cage to keep them from running away. And making them feel bad about using Fedora really isn’t going to get them to spend money on your paid version; it’s going to send them somewhere else.
One change at RH between V3 and V4 was handling its partner relationships. It developed what’s basically an open-source training program where companies could send people to work at RH and learn how the open-source development process works.
I’m glad to hear that RH has continued to take the SELinux devirginization on its own hands in the corporate side as well as in Fedora. That way the rest of us can just wait until things are working before we include in our standard offerings. =)
The biggest point in this story:
“Upstream, Upstream, Upstream”-this became the mantra among our kernel team throughout the entire duration of Red Hat Enterprise Linux v.4 construction.
Why upstream? The reasons are obvious, once you hear them:
- More eyes on the code
- More people compiling and running the code
- Shift the maintenance burden elsewhere
- This is a subpoint of the last one: No need to forward-port patches
This has become my mantra in recent months too, with Gentoo’s X work.
Perhaps The Cathedral and the Bazaar should become required reading for all corporate distro employees, so they can understand why it’s a superior model, even if it initially seems like more work to migrate to it.
I do find it more than a little odd that RH rejected those proprietary add-ons in the V3 story, but in this one, it was happy to bundle proprietary add-ins like RealPlayer, Acrobat Reader, Flash, Citrix and a JRE. Perhaps it should instead be working on free alternatives to those still lacking a good, updated one, and shipping those instead?
Striving for greatness 