FRAMINGHAM (01/28/2000) - There's an absolutely critical new skill that most IT staffs sorely lack. It's a skill that clearly separates successful dot-coms from e-commerce wanna-bes. It's also a skill that doesn't really have a name yet. For now, let's just call it Web-application scalability engineering. It's part science and part art, and it's crucial to dealing with the new economics of e-commerce.
I know what you're thinking: "We have Java programmers. We have Unix systems administrators. They know how to build systems that scale." But do they? Or do they just know how to throw hardware at the problem?
I've seen dot-coms achieve 160 times the capacity of a corporate Web site with a budget one-fifth as big. How do they do it? They're sure not gonna tell you, because it's at the core of their competitive advantage.
But what I can tell you is the dot-coms have some smart technical people on their teams who take a very cross-disciplinary approach to achieving scalability. These people understand networking issues like Internet service provider peering (how their providers interconnect with other ISPs) and payload-to-header ratios (a factor that determines how efficiently they use their available bandwidth). They understand systems issues like processor utilization and caching. They understand software engineering issues like database connection pooling (which reduces the strain on back-end resources) and intelligent agents. Their holistic view of how these components affect a customer's ability to get a fast response when they click on a Web-page button makes them formidable competitors.
These new IT alchemists know how to stretch a budget, too. They have zero tolerance for software licensing schemes that penalize them for success. They'd rather write a database from scratch than shell out six figures for the privilege of running a vendor's solution across 48 commodity Intel servers.
They understand that scale isn't just about MIPS and megabits per second. It's about dollars and cents.
You see, you can't achieve scale when you're still taking a stovepipe approach to the problem, divvying up infrastructure responsibilities among systems administrators, database administrators and application developers. The new breed of technosavant is a polymath who understands that to achieve end-to-end performance you need an end-to-end perspective. That end-to-end mentality is something to which most IT shops still only give lip service. But the uber-geeks who ride shotgun on the Web's top sites have it scoped. If you don't think your company's success hinges on Web-application scalability engineering, just look at outfits like DoubleClick, Amazon.com and Motley Fool. They've been able to rise as far and as fast as they have because they've cracked the code of scalability. And they did it before the cash started pouring in.
The Web operates under a set of technical and economic principles vastly different from the world of client/server application development. At the heart of that difference is scale. Once your site starts attracting heavy traffic, scalability is what makes the difference between revenue and ridicule.
Recognize it as a distinct discipline. Cultivate it among the members of your staff with the right competencies and disposition. Or prepare yourself to become the next Web scandal. Those are about your only choices in the mega-hit per second world of the Web.