Cover Story: Then and now . . . do IT analysts ever get it right?

They're considered the top guns in the IT pundit market. And many organisations depend on their forecasts before committing to top dollar IT investments. But there's no guarantee that IT analysts will always get it right.

This month's cover story looks back at what some major analysts have predicted and just how close -- or far -- they've come to hitting the mark!

Triumphant music swells in a darkened amphitheatre. Galaxies of facts sparkle to life on overhead audio visual screens as a man in a superbly cut suit strides to the podium.

For the next hour he weaves masses of detail into analytical tapestries that flutter toward the future. His audience of hard-headed technology managers is paying top dollar for advice on navigating their organisations safely through the dangerous rapids of IT. They sit spellbound while scenarios, facts and predictions dart at them like swarms of brightly coloured tropical fish.

At the end they applaud thunderously and queue up to ask personalised questions of the visionary on stage.

It is a scene played out many times a year at customer briefing extravaganzas staged by the oracular organisations who analyse and predict technology trends for a living. Top guns in the pundit market include GartnerGroup, International Data Corp, Forrester Research, Meta Group and Yankee Group.

Their yearly showcase events are the computer industry's version of religious revival meetings. Like their evangelical counterparts, these apostles of infotech operate in a climate of received truth. They rarely admit to being wrong. But when the presentation pyrotechnics are peeled away and their forecasts are checked in the cold light of reality, just how well do they hold up?

BLIND SPOTS

To find out, we travelled back in time to unearth what some of the top-tier market research and analysis organisations said a few years ago about the shape of the IT world circa 1998.

We isolated predictions made at various annual briefing events in the early to mid-'90s about Windows NT, Java, enterprise applications and application development, thin clients, platforms and operating systems, outsourcing and the Internet.

The results provide food for thought to technology managers who depend on the prognosticators to help them rig safety nets under their planned IT investments.

On issues that might be called a traditional part of the technology spectrum -- mainframe and storage trends spring to mind -- the top pundit groups generally did an outstanding job. Their record was less impressive in new or rapidly evolving areas where the baselines were not so clear-cut.

And they ran into real problems with Big Hit events, large-scale occurrences whose timing is tricky to forecast but whose effects are huge.

The 1997/98 Asian meltdown was a Big Hit event whose imminent arrival was missed by everybody from the World Bank on down and whose repercussions made a hash of IT market predictions for that part of the globe.

Other Big Hit incidents were the merger of Compaq and Digital Equipment Corp and, more recently, of Netscape and AOL. In each case, the participants and timing caught IT pundits by surprise and both acquisitions had the potential to change life dramatically for customers.

Yet another variety of Big Hit was the creation in 1991 of the World Wide Web's HTTP protocol and the subsequent explosion of browser-backed intranets and extranets. They proved a blind spot on the radar screens of many prognosticators.

NUMBERS GAME

The technology soothsayers are paid not only to highlight significant trends in the confused jumble of marketplace claims but to put numbers on those trends.

Looking at predictions made in the mid-'90s about Windows NT, all the pundits were correct in calling NT's general direction (rapidly upward). However it was a different matter when it came to estimating the strength of the move.

IDC's 1996 annual Australian briefing correctly forecast that Windows NT would make inroads at the low-end server market yet underestimated how badly it would erode Novell's market share.

IDC's briefing notes that year forecast Novell NetWare server shipments would grow 25 per cent in 1997. In fact they fell 9.4 per cent, a drop that was a direct result of NT's surging popularity. IDC in 1996 believed new shipments of NT would trail NetWare by 150,000 units worldwide in 1997. The reality was that NT shipments passed NetWare for the first time that year.

Such miscalculations highlight the fundamental dilemma facing the forecasting brigade. They can achieve 100 per cent hit rates by stating the obvious in carefully general terms or they can be 100 per cent wrong by aiming at over-ambitious levels of detail in complex and rapidly evolving situations.

An internal study conducted by GartnerGroup in 1997 evaluated 400 strategic planning assumptions generated in 1992 and produced a self-assessed accuracy rate of a comfortable 71 per cent.

To Gartner that indicated it was striking the right balance between stating the obvious and making the tougher long-term calls. The study found the best performance (73 per cent hit rate) in the area of platforms and operating systems. Lowest (68 per cent) was in enterprise application and application development.

The in-house evaluation highlighted how choosing the right timeline presents a perennial pitfall to technology forecasters. In 1991, Gartner predicted most leading CASE tools would incorporate or support Object Oriented (OO) technology by 1994. In fact most of the leading tools didn't build in OO analysis, design, construction and deployment support until 1998.

By its own measure, GartnerGroup conceded it badly overestimated the speed at which CASE vendors would roll out their migration paths to OO and components.

INTERNET

The rise and rise of the Internet initially caught most professional pundit groups on the hop -- a situation that may be repeating itself with open source code today.

Although they now devote acres of analysis to intranets and electronic commerce, some of the biggest research and analysis organisations were caught napping when the World Wide Web took off like a supernova around 1993. Meta Group's agenda summary for its 1993 METAmorphosis conference, for example, didn't mention the World Wide Web in its blurb on global networking strategies.

The compressed life cycle of Internet products increased the embarrassment potential for analysts. By the time a trend became noticeable, it could already be sliding towards obscurity (hands up, all those still betting on push technology).

Even Internet technologies with proven legs, such as Java, can be a forecaster's nightmare. Three years ago, GartnerGroup hazarded a prediction that by early 1998, Java would become the de facto Internet "machine language" for the development of Web pages with executable content. It also said 30 per cent of Web sites would have executable content by 1999.

Neither prediction rings true in the opinion of former Australian Java user group president John Wood.

He says the browser wars ended in Microsoft's favour before either Microsoft or Netscape fully implemented Java version 1.1 which would have accelerated the execution of Java content to a degree that users found acceptable. So many Web developers still aren't building Java applications into their offerings.

In addition, most Web sites are still being used as simple billboards with a resurgence of fast, efficient content and less emphasis on flashy Java-supported animation.

On the other hand, GartnerGroup was on the money with its prediction in 1996 that the largest obstacles to commercial development of the Internet would be resolved by 1998.

WINS and LOSSES

As noted, open source code, also known as freeware, may be the latest analysts' blind spot. Linux didn't rate a mention in the printed briefing papers at IDC's 1996 forum. Yet it rocketed from seventh to fourth place in operating systems between 1996 and 1997 and the open source movement made more gains in 1998 as IBM, Sun Microsystems, Netscape and a host of other vendors clambered aboard the bandwagon.

GartnerGroup proved more prescient when it used its 1996 customer briefing to puncture some of the thin client and Network Computer (NC) hype. At that time Gartner said users would resist thin clients in favor of the traditional fat PC for the next few years at least and opined that Network Computers (NCs) would not prove popular for more than a small fraction of office workers. Its judgement was justified by developments in 1998 which included concessions by the prime NC booster, Oracle CEO Larry Ellison, that NCs hadn't taken off.

Dataquest put NC shipments in 1998 at around 450,000 units, well below previous expectations. PC prices that plummeted in response to the threat of a $US500 NC were one reason the NC bandwagon stalled. Another was the expected emergence of the Windows Terminal.

Outsourcing was an area where pundits can claim to have earned their keep. Gartner's general analysis in the early 1990s of the broad drivers behind outsourcing and the evolution of the trend into selective outsourcing have been largely borne out by events.

Five years later, it is still difficult to fault Gartner's 1994 assessment that the effectiveness of an IS shop will be judged "not only on its ability to manage traditional projects but on its ability to build a flexible infrastructure that can absorb and optimise resources that are changing very dynamically. IS staffs that do not build (such a structure) will find their positions outsourced to companies that can."

Sometimes forecasts that are correct in their general thrust lack the telling detail to be truly useful. So it was with Gartner's warning at its 1996 briefing that more than half the vendors in the market would cease operations or be acquired within five years. Any IS shop contemplating an upgrade of its DEC platforms at that time couldn't say the warning wasn't issued but the value of the advice was severely diluted by its vagueness.

Much crisper and more clear-cut was the foresight supplied in areas like mainframe and mass storage trends plus early identification of the shift to client/server architectures.

Gartner's prediction in 1996 of an accelerating trend towards off-the-shelf purchases of large integrated applications may fall into the category of stating the obvious but has been amply borne out by the ERP vendors' soaring revenues.

In the networking arena, on the other hand, its 1992 expectation that ATM-to-the-desktop would be the norm for large enterprises by 1997 proved over-optimistic. Gartner now believes the cost of ATM will confine it largely to the backbone and WAN for at least another few years.

ASIAN FLU

The Asian economic flu of 1997/98 proved fatal to IDC's forward estimates of world and Asia Pacific IT market outlooks made in 1996. At that time, IDC predicted the world semiconductor market would touch $US300 billion in two years. The 1998 figure actually came in closer to $125 billion due to the Asian crisis and sluggish first half PC demand. The 16 per cent growth IDC called for in Asia Pacific's IT spending during the 1997/98 period turned into a drop of 26 per cent which in real terms was probably closer to 50 per cent thanks to dollar devaluations of the Asian currencies.

The lesson is plain enough for IT companies gearing up for overseas expansion: take any long-term regional economic forecasts with a large dose of scepticism.

INTESTINES

Clients rely on the market analysis organisations to reduce the anxiety levels associated with making strategic, big dollar decisions about complicated, fast-moving technology environments.

Once submerged in the big conference torrent of statistics, analytical legerdemain and polished presentational skills deployed by the professional prognosticators, however, information overload can become a problem in its own right.

Of course paying clients didn't have to rely on one-shot injections at the annual conferences. They can pursue issues via fine-grained discussions in personal interviews with analysts or access a continuously updated stream of position papers.

Even with the benefit of looking back through the prism of 1998 at briefing notes handed out by Gartner and IDC in previous years, it is difficult to rate many predictions as either clear failures or total successes.

The language is usually couched in carefully conservative phrases which can require clients to read between the lines.

Thus you would seek in vain a few years ago to find a prediction baldly stating that token ring was a dead-end and long live Ethernet. Instead, you'd have found predictions that token ring's market share would continue to drop.

Overall, the analyst organisations we concentrated on -- GartnerGroup, IDC and to a much lesser extent Meta Group -- played well on the big points.

They give customers their money's worth in terms of outlining major trends and picking inflection points. But as the predictions descend to more detail or longer time frames, customers are well advised to reduce their quotient of belief.

On a final, upbeat point, it is true to say that the art of prophesy has racked up at least one major advance since its pioneering days in Greece and Rome.

That is, there is no doubt the sound-and-light show at any IT forecaster's conference is a vast improvement over watching the Oracle at Delphi pick through animal intestines.

Join the newsletter!

Error: Please check your email address.

More about AOLAudio VisualCompaqDataquestForrester ResearchGartnerIBM AustraliaIDC AustraliaInfotechMeta GroupMicrosoftNovellObject OrientedOraclePrismSun MicrosystemsWorld BankYankee Group

Show Comments