ONE OF THE KEYS TO benchmarking, consultants say, is to see it as only the first step in a learning process. As these case studies below reflect, IT managers facing a wide range of organizational challenges can use benchmarks in many different ways. It is all part of measuring efficiency and keeping their internal customers satisfied.
Competing with outsourcers
Mobil's data center is under constant pressure to perform. Because hardware prices have dropped, industrywide data center costs have fallen about 15 to 20 percent a year, making efficiency a moving target. Regular measurement and improvement are essential.
"We charge out for services, and if we look expensive to our internal customers, they'll be wanting to outsource," says Rich O'Connor, global systems support manager at Mobil, in Dallas. (Mobil recently merged with Exxon to become ExxonMobil.) Some benchmarking efforts have been instigated by the corporation's chief financial officer, who wanted to know whether the organization could save money by outsourcing the data center. "We eventually elected not to outsource," O'Connor says, but it's because the data center has made gradual improvements each year. "I truly believe that the benchmarking we do is part of a continuous improvement process here and not something we do to get a pat on the back."
At the beginning of each year, the Mobil data center negotiates service-level agreements with each business unit, so it's clear what new systems or services they will need and what reliability levels are to be met. Then the IT staff keeps an internal scorecard of how effective it is in meeting those criteria.
The data center, which has 80 employees, assigns employees to categories following a model supplied by consulting company Compass America so that it can make "apples to apples" comparisons with other data centers. Compass puts Mobil in a group with five other data centers to compare the high, low, and mean on the thousands of benchmarks it runs, O'Connor says. These numbers are also pored over with the CIO's staff to help identify areas that need improvement.
Metrics to improve communications
In 1996 the Commonwealth of Kentucky instigated Empower Kentucky, an effort to use information technology more efficiently. Improving the state's IT performance and its relationships with employee users are key elements of this overhaul, says Bob Morgan, an IT systems engineer in the Division of Planning & Architecture of the Governor's Office for Technology, in Frankfort.
"At the end of the day, we are a service organization, and the PC is our pick and shovel," Morgan says. "It's how we approve land-use requests or prepare for the surfacing of a road." The state Department of Information Services (DIS) is preparing service-level agreements to define partnerships with its customers, says Morgan. The goal is to provide a base of open communications that will result in continuous service improvement.
Morgan says Kentucky will combine internal metrics with customer-focused outcome measures. These will include personal interviews and debriefings with customer CIOs and executives, periodic interviews with customer management, and occasional benchmarks for verification.
The state has created a public Web site (http://cdc.state.ky.us/metrics) that will post its metrics for all to see.
Historically DIS has had difficulty getting good feedback from internal customers. It has done project closure surveys, but "very few were completed," Morgan says, "and we had reason to doubt the veracity of those that were.
People candy-coat things and say they were wonderful, when in fact we know they weren't wonderful."
"This is where service-level agreements can come into play," Morgan says. "They can be one of the cornerstones of continuous improvement, not as a punitive device, but as a vehicle of communications."
When benchmarks don't help
For some, gathering performance data isn't enough. If faced with shrinking budgets, talk of performance metrics can fall on deaf ears. Just ask James Remedios, CIO for the City and County of Honolulu.
In Hawaii, the state and city government's budgets have been cut five years in a row. Then, the City's Department of Information Technology had 122 full-time staffers to cope with 4,000 employee users. Now it has only 104. "The mayor talks about how information technology is going to carry us into the future, but it's not happening," Remedios says. IT regularly loses budget battles to the city's more pressing needs.
Although he does survey users, Remedios says getting realistic feedback is difficult. The comments are rosy because nobody wants to "talk stink" about anybody else, he says, especially in Hawaii's business culture, which is considered less confrontational than that on the mainland. "You have to measure these things yourself," Remedios says.
Remedios and his staff have developed internal measures of effectiveness, including ways to track how quickly the department responds to help requests.
The metrics point out only what he already knows: "I could use 27 more people right now," he says. "Our help desk is a real problem. And I have only three guys handling networks for the whole island [of Oahu] now. It's a credit to them that we keep it up, but I need more network support people. Of course, our overall performance is going to suffer."
Remedios also has a contract with Gartner Group, which gives him comparable numbers for other cities and "best practices" examples. But, he says, "they don't tell me anything I don't already know. We know what we should be doing, but unless we invest more, we're never going to get out of the swamp."