After a disastrous infatuation with computer-aided software engineering (CASE) tools in the early '90s, most IT organizations realized they had put too much faith in "silver bullet" tools, and decided instead to emphasize a combination of people and processes. At one extreme, we have "heavy" processes combined with "light" (inexperienced) people; at the other, we have "light" processes and "heavy" (experienced) people.
The latter scenario is exemplified by the Extreme Programming (XP) approach, but is also known by a number of other buzzwords collectively referred to as "agile" methodologies. At its core, the agile/XP methodology movement consists of pragmatic principles that encourage developers and end users to develop systems in an environment in which requirements are constantly changing and collaboration among the individuals is emphasized over formal processes, comprehensive documentation and overly legalistic contracts.
Thus far, most XP/agile projects have been relatively small for example, a half-dozen developers interacting continuously with a half-dozen users over three to six months. And that has led to a dismissive response from senior IT executives: "XP may be useful on little toy projects, but it certainly won't work on the big enterprisewide projects that we're contemplating." I'm not convinced that's valid, and I believe that a big challenge for IT organizations in the coming years will be finding ways to apply the XP/agile concepts on a large scale.
Here's an example of a difference between small, short XP projects and large, long-lasting projects: the "death-march" culture. A handful of highly motivated workaholics can work 80 hours per week for a few months if they choose to, and the organization's management culture may glorify such behavior. But it's simply not possible when 300 people are working on a massive system that will take five years to finish; while some overtime may be necessary, common sense suggests that the overall project schedule must be based on a sustainable effort by everyone concerned.
The biggest challenge for large-scale XP/agile projects is communication. I asked one senior IT executive to describe his company's development process. He said, "We put five developers in a room and let them shout at each other." That may be an oversimplification, but XP/agile projects put a great deal of emphasis on face-to-face interactions between developers and users, with frequently delivered prototypes substituting for written documents. It's easy to imagine that with five developers and five users, but what about 50 developers and 50 users? Or 500 developers and 500 users? The obvious strategy is to break the large project into smaller, semi-independent pieces; but if the entire system has to function as a tightly integrated whole, and if the interfaces between the various subsystems are changing continuously, communications can be a challenge.
One solution involves the concept of emergent behavior, which XP/agile gurus like Jim Highsmith have been advocating. In the IT world, perhaps the best example is the Internet. As Greg Papadopoulos suggested at Sun Microsystems Inc.'s JavaOne conference last month, "You don't reboot the Internet; you live with it. It's fundamentally unknowable and unplannable." And consider a non-IT metaphor: Planned cities like Brasilia and Canberra often turn out to be sterile and boring; but vibrant cities like New York and London evolve and grow in a natural, organic fashion, based on actual usage by their populations.
The challenge for IT will be to learn how to manage the creation of a New York or London without succumbing to the bureaucracy that creates a Brasilia or a Canberra. I don't know if we'll succeed, but the most interesting and pragmatic strategies are likely to come from the XP community. If you're an IT manager, you owe it to yourself to begin paying close attention to this group of leading-edge thinkers.
Ed Yourdon is editor of Cutter IT Journal, published by Cutter Consortium in Arlington, Mass. Contact him throughwww.yourdon.com.