As the number of Linux kernel contributors continues to grow, core developers are finding themselves mostly managing and checking, not coding, said Greg Kroah-Hartman, maintainer of USB and PCI support in Linux and co-author of Linux Device Drivers, in a talk at the Linux Symposium in Ottawa last week.
In the latest kernel release, the most active 30 developers authored only 30 percent of the changes, while two years ago, the top 20 developers did 80 percent of the changes, he said. Kroah-Hartman himself is now doing more code reviewing than coding. "That's all I do, is read patches these days," he said.
In theory, the kernel development process would involve changes, or patches, moving from the original author through a file or driver maintainer, to the maintainer of a major subsystem such as PCI or SCSI, to Andrew Morton for testing and finally to Linus Torvalds for a kernel release. But, Kroah-Hartman said, "I tried graphing that. That's not what happens. It's a mess. There's routing all over the place."
A graph of all the developers involved in the upcoming 2.6.22 release, and the relationships of who reviewed whose patches, extends to a 40-foot-long printout with names in tiny type. The graph is on display at the Ottawa event.
The new "mess" results in innovative new features getting integrated into Linux distributions much more quickly, says Jonathan Corbet, author of the camera driver for One Laptop Per Child and another co-author of Linux Device Drivers. Previously, when developers maintained both "stable" and "development" kernels, it could have been two to three years before a feature made it from development to mainstream users.
Today, by contrast, the newly released Fedora 7 distribution has the power-saving tickless kernel functionality, which came out in the 2.6.21 kernel in April.
Enterprise Linux distributions that pick a single kernel.org release and maintain it for five to seven years are another reason for the "mess." Instead of waiting for a stable upstream release and then modifying it to include new functionality from development kernels, an enterprise distribution can QA and support any of the 2.6 releases, which come out every 2.5 months. "The patch loads carried by the distributors have shrunk quite a bit," Corbet said.
Kernel release 2.6.11, released in March 2005, had 475 developers, and the upcoming 2.6.22 release, scheduled for July, has 920 developers so far, Kroah-Hartman said. The number of kernel changes is also growing. While 2.6.11 had about two changes per hour, 2.6.22 is at four changes per hour. And even when Linus Torvalds took a month off to write the git revision control system in the spring of 2005, the rate of change was faster than the entire 2.5 "development" tree. "We are going up. We are going faster than we have before. This scares the heck out of traditional computer scientists," Kroah-Hartman said.
Despite Red Hat's dominance of commercial Linux sales, the spread of actual development contributions is much more diverse, with Novell, where Kroah-Hartman works, punching above its weight at 9.7 percent of kernel contributions to Red Hat's 11.8 percent. The next three companies in the rankings are IBM, Intel and SGI, which has installed a 4096-processor supercomputer, the largest single system image Linux box yet.
But just above SGI in the listings are "amateurs." Although the great independent Linux hacker has been thought extinct, a casualty of corporate recruiting, known amateurs still count for 3.9 percent of kernel changes. Reflecting the large number of tiny changes, the top category is still "unknown", but everyone with 10 or more changes in the kernel is accounted for, Kroah-Hartman said.
The large contributor base, huge codebase (8.2 million lines), and rapid change means that it's impossible to maintain quality kernel-space code for Linux outside the development process, Kroah-Hartman said. "You're never going to be able to catch up. It's just physically impossible," he said. "If your company isn't showing up here and you need to affect how Linux is developing in the future, you need to get involved," he added.
An audience member asked about kernel size comparisons to Microsoft's Vista operating system, but the two projects aren't directly comparable, because Linux maintains its device drivers "in the tree" and available for all developers to see. Fifty-two percent of the size of the kernel is in drivers, and less than 5 percent is "core", Kroah-Hartman says. However, changes happen in the core at the same rate as in other parts of the kernel, he said. APIs and core data structures change regularly.
With the rate of change growing, what about bugs? Kroah-Hartman said he monitors bug reports from several distributions, and added, "I've seen a flat rate over the past two years." Constant bugs and growing changes implies fewer bugs per change, but there is always room for improvement. Linux needs a person to coordinate bug tracking, Corbet said, and Google has offered to hire someone to do the job.
Users can't really blame the kernel hackers for moving stuff around. Many users and vendors are using Linux for diverse requirements, from high-performance computing to mobile phones, and users' hardware and application needs drive kernel changes.
"We are changing based on external stimuli," Kroah-Hartman said, adding that Linux is the only operating system that runs on very small embedded systems and on large NUMA systems from SGI and others. "That's an architecture scalability that nobody has ever achieved. We're achieving that thanks to our rate of change," he said.
In the past two years, 3200 developers have contributed at least one patch, with half contributing two or more, one quarter contributing three or more, and so on, in a nice "long tail" exponential curve. Every patch will have at least one author and one reviewer. "It is a web of trust. Linus trusts 10 to 15 people, I trust 10 to 15 people," Kroah-Hartman said. "I trust that they'll be around to fix the problem. And that's more important than getting it right in the first place."