The Big Picture

The kernel developers use a loosely time-based release process, with a new major kernel release happening every two or three months. The recent release history looks like this: 

2.6.26 July 13, 2008
2.6.25 April 16, 2008
2.6.24 January 24, 2008
2.6.23 October 9, 2007
2.6.22 July 8, 2007
2.6.21 April 25, 2007
2.6.20 February 4, 2007

Every 2.6.x release is a major kernel release with new features, internal API changes, and more. A typical 2.6 release can contain over 10,000 changesets with changes to several hundred thousand lines of code. 2.6 is thus the leading edge of Linux kernel development; the kernel uses a rolling development model which is continually integrating major changes.

A relatively straightforward discipline is followed with regard to the merging of patches for each release. At the beginning of each development cycle, the „merge window“ is said to be open. At that time, code which is deemed to be sufficiently stable (and which is accepted by the development community) is merged into the mainline kernel. The bulk of changes for a new development cycle (and all of the major changes) will be merged during this time, at a rate approaching 1,000 changes („patches,“ or „changesets“) per day.

(As an aside, it is worth noting that the changes integrated during the merge window do not come out of thin air; they have been collected, tested, and staged ahead of time. How that process works will be described in detail later on).

The merge window lasts for two weeks. At the end of this time, Linus Torvalds will declare that the window is closed and release the first of the „rc“ kernels. For the kernel which is destined to be 2.6.26, for example, the release which happens at the end of the merge window will be called 2.6.26-rc1. The -rc1 release is the signal that the time to merge new features has passed, and that the time to stabilize the next kernel has begun.

Over the next six to ten weeks, only patches which fix problems should be submitted to the mainline. On occasion a more significant change will be allowed, but such occasions are rare; developers who try to merge new features outside of the merge window tend to get an unfriendly reception. As a general rule, if you miss the merge window for a given feature, the best thing to do is to wait for the next development cycle. (An occasional exception is made for drivers for previously-unsupported hardware; if they touch no in-tree code, they cannot cause regressions and should be safe to add at any time).

As fixes make their way into the mainline, the patch rate will slow over time. Linus releases new -rc kernels about once a week; a normal series will get up to somewhere between -rc6 and -rc9 before the kernel is considered to be sufficiently stable and the final 2.6.x release is made. At that point the whole process starts over again.

As an example, here is how the 2.6.25 development cycle went (all dates in 2008): 

January 24 2.6.24 stable release
February 10 2.6.25-rc1, merge window closes
February 15 2.6.25-rc2
February 24 2.6.25-rc3
March 4 2.6.25-rc4
March 9 2.6.25-rc5
March 16 2.6.25-rc6
March 25 2.6.25-rc7
April 1 2.6.25-rc8
April 11 2.6.25-rc9
April 16 2.6.25 stable release

How do the developers decide when to close the development cycle and create the stable release? The most significant metric used is the list of regressions from previous releases. No bugs are welcome, but those which break systems which worked in the past are considered to be especially serious. For this reason, patches which cause regressions are looked upon unfavorably and are quite likely to be reverted during the stabilization period.

The developers‘ goal is to fix all known regressions before the stable release is made. In the real world, this kind of perfection is hard to achieve; there are just too many variables in a project of this size. There comes a point where delaying the final release just makes the problem worse; the pile of changes waiting for the next merge window will grow larger, creating even more regressions the next time around. So most 2.6.x kernels go out with a handful of known regressions though, hopefully, none of them are serious.

Once a stable release is made, its ongoing maintenance is passed off to the „stable team,“ currently comprised of Greg Kroah-Hartman and Chris Wright. The stable team will release occasional updates to the stable release using the 2.6.x.y numbering scheme. To be considered for an update release, a patch must (1) fix a significant bug, and (2) already be merged into the mainline for the next development kernel. Continuing our 2.6.25 example, the history (as of this writing) is: 

May 1 2.6.25.1
May 6 2.6.25.2
May 9 2.6.25.3
May 15 2.6.25.4
June 7 2.6.25.5
June 9 2.6.25.6
June 16 2.6.25.7
June 21 2.6.25.8
June 24 2.6.25.9

Stable updates for a given kernel are made for approximately six months; after that, the maintenance of stable releases is solely the responsibility of the distributors which have shipped that particular kernel.