I have always been amused by simplistic methods offered to CIOs to answer complex questions about the health of IT.
One recent attempt is particularly noteworthy because it received widespread publicity and reputable endorsements. Last fall, a journal for CIOs published a five-page feature, advertised as a diagnostic tool, with the ambitious title "How to Improve IT/ Business Alignment." Introductory explanations claimed that this technique was successfully tested with more than 25 Fortune 500 companies and used in studies sponsored by the Society for Information Management and The Conference Board.
It required the placement of check marks on a grid consisting of five columns and 38 lines representing 38 questions. A check mark in the first column was worth one point, a mark in the second column two points, and so forth, with five points for each mark in the fifth column. A respondent could score anywhere from 38 to 190 points, with a relatively high score signaling a high level of IT alignment with the rest of the business.
The proponents' assumption was that the higher the score, the stronger the alignment, and therefore the more benefits a business would see, which would ultimately be reflected in its financial performance. For instance, if a CIO reported to the CFO, that would be worth one point. If the CIO reported to the CEO, that would be five points.
"Rarely measuring the value of IT investments" was worth one point, but using a balanced scorecard approach awarded you five points. Other characteristics that were scored included service-level agreements, how projects are prioritized and architectural integration.
The publicity given to this methodology and its claims heightened my curiosity about its value. As a scorekeeper of corporate profitability and IT spending, I offered to test the tool's validity. The researchers graciously provided me with each corporation's individual score (subject to an agreement to not reveal the identities of the firms or researchers).
My findings? After I compared the alignment scores with the actual financial results delivered by the firms that were used to test the methodology, I found absolutely no relationship between the overall score and other independent measures of financial results. There were high-scoring firms that were losing money, and there were low-scoring firms that delivered exceptional profits. So the methodology failed in perhaps its most important test of relevance to a board of directors.
I view that as an example of contemporary astrological practice as applied to assessing IT excellence. Astrology, in which I take an active interest, takes factual observations about stars, planets and the calendar and converts them - by unverifiable methods - into prophecies about events on Earth. I would classify any method for cobbling together descriptions about IT-related practices into predictions about IT excellence as being like astrology rather than an application of reproducible analysis that can help IT.
I have seen many examples of methodologies that follow similar patterns. Some are relatively modest and perhaps involve filling out a grid. Others are ambitious, often asking that more than 150 line items be scored via elaborate schema. This includes most of the measurements of an organization's "maturity level." The widely applied balanced scorecards disguise their astrological origins by arbitrary choices of ratings criteria. For instance, one requirement to qualify a firm's ranking of its computing innovation attributed 20% of that score to whether it installed the latest version of Windows on all desktops!
All of these methods measure criteria that are mostly unrelated to top executive concerns, which are primarily economic. In other words, CEOs couldn't care less about conformity with what academia claims are desirable behaviors. What they do care about is increasing profitability and whether IT is helping to do that.
As IT management and budgets come under increasing scrutiny today, don't bank on scoring well using astrologylike methods. Proving that dollars saved and earned will add to the bottom line will count for more.
- Paul A. Strassmann (firstname.lastname@example.org) believes that astrologylike methods are popular ways to measure the health of IT because they're cheap, don't require much effort and avoid accountability.