How do you quantify quality? It's something PacifiCorp's 300 IT employees wrestle with daily, as they are beholden to specific service-level benchmarks and regularly comparing themselves to outsourcing options.
"What we are doing is trying to pick some areas where we want to be able to compare ourselves to the industry at large," says Tim Meier, CIO at PacifiCorp, in Portland, Oregon. All of the electric utility's IT managers strive to find industry data pertinent to their area. "We try to measure [ourselves] in a comparable manner so we can put our performance in perspective with the industry."
Meier's team recognizes that the internal IT function competes with other service provisioning alternatives such as outsourcing options. "To the extent that we can deliver good comparable service at best value, the internal IT [staff] stays in business," he says. "If not, we seek to partner with someone that can provide best value."
A Way to Measure Your Work
Almost every IT organization performs some kind of internal performance measurement. This "benchmarking" has become an increasingly important and standard tool for IT departments to help justify expenditures and gauge internal customer satisfaction.
But there are two different types of benchmarking, and they measure very different things.
Dave Burkett, president of IT consultancy Compass America, says that 90 percent of his Reston, Virginia-based company's customers are concerned with measuring efficiency, whereas only 10 percent focus on IT effectiveness. Measuring efficiency involves some basic metrics such as network uptime or help desk response that are then compared with other companies as well as internally over time, while measuring effectiveness might involve comparing practices with other IT departments and trying to understand how much IT contributes to the basic strategy of the business.
Basic metrics prove valuable in highlighting disparities from industry norms, but they don't necessarily point out if you have a problem, nor how to address it, notes Steven Gordon, associate professor of information systems at Babson College, in Babson Park, Massachusetts. For instance, Gordon says, a CIO who finds that a relatively low percentage of problems are resolved on the first call to the help desk might think his or her staff is poorly trained, or that the problems the help desk is trying to solve are complex and hard to diagnose.
Each possibility implies a different solution and requires more effort than the simple benchmark to diagnose, Gordon says. Companies spending more than the norm on IT may be overstaffed or they may be employing IT much more strategically than competitors, so a larger IT spending/revenue ratio is neither a positive nor a negative signal in and of itself. "Once you've identified a disparity, that's when you should do a more thorough internal analysis," Gordon says.
Benchmarking to measure effectiveness, or "best practice" benchmarking, is much rarer, in part because it's hard to do. It can be difficult to find comparable IT organizations or to ensure that another company's "best practice" can be translated to your company. Also, IT executives who would like to engage in sharing information with their peers about a process -- such as recruiting and retention -- often feel too strapped for time to attend meetings and conferences, says Gordon, who has moderated a CIO "best practice" roundtable.
"Many are happy to share this information and say they find it valuable, but then they won't take the time to do it."
It's also not part of the IT managers' mind-set to see themselves as a strategic asset, Gordon adds. "IT departments tend to perceive themselves as service organizations. If you ask a CIO they'll say they do not drive the business, the business drives them. For instance, if the company is going to establish a new e-commerce initiative, they believe it will fail if the idea comes from the CIO."
Many IT execs are too focused on service delivery to make the time to examine their practices, says Sheila Smith, managing partner at Omega Point Consulting in Palm Beach Gardens, Florida. But some realize that process improvement is essential. "One CIO we worked with told his direct reports that he felt that 50 percent of their job was in improving how the IT organization does its work," she says.
If an IT department does examine and compare its processes, it may be part of an organizationwide "total quality management" effort. "Business process measurement is the future of benchmarking," Burkett says, "but it's kind of new." People used to see their own processes as unique, but the widespread use of ERP (enterprise resource planning) software has changed that perception, he says. Now people are starting to realize that their processes are reasonably generic and can be measured and compared. Burkett says that once a company has completed a set of benchmarks, it may find obvious problems that weren't readily apparent. But they usually aren't funding deficits, nor departments that are overstaffed. Instead it is common, he says, that "deployment of resources is not at all optimized. Often they've installed software to help them with a certain process, and yet they're still using workers to do that process the old way. With help from an outside, unbiased source, they can be pleasantly surprised that they can move workers to other areas without any degradation in performance."
How to Assess
So how do most IT organizations assess themselves? "The best ones do it in multiple ways," Smith says. "The intent should be to take a snapshot and see where the weaknesses are and make organizational improvements to address the gaps." Often there is a focus on customer surveys, which can help counteract negative impressions about the IT department's performance, she adds.
Customer satisfaction surveys are usually done annually, Smith says, and are focused on the products and services for which IT is responsible, as well as general service characteristics such as responsiveness. End-of-project surveys are directed at a specific customer base that was involved in the project, providing more timely feedback to correct performance issues.
Customer expectation interviews help managers understand what users expect in terms of leadership skills in IT, Smith says. For instance, surveys that Omega Point prepares for customers ask, "In light of the projects that you may have coming up in the new year, what do you expect of the IT leaders to help make those projects successful? What new competencies, behaviors, or qualities will IS leaders need to develop? What competencies, behaviors, or qualities will IS leaders need to enhance?"
Some departments also do "lessons learned" sessions with customers -- either after a specific effort or periodically around the service IT is providing.
Also, IT managers can ask for "360-degree feedback," Smith says. This involves individual performance evaluations completed by the person's supervisor, subordinates, peers, and customers. It provides specific information about the person's strengths and areas for development. "This is great feedback on your one-to-one performance," she says.
Omega Point also helps IT organizations set up scorecards with measurements of four dimensions: those areas important to the management of the company; areas important to IT customers; areas important in terms of the efficiency and effectiveness of internal IT processes; and areas that indicate whether IT is positioned for the future, usually learning and growth issues.
Another concept that most people usually associate with outsourcing is service level agreements. Many IT organizations routinely establish "contractual" agreements with their internal customers about levels of service they will provide throughout the following year. This gives both sides clear targets for assessment of whether IT is functioning well.
Hitting the Numbers
Tim Meier used benchmarking as part of an overall plan to restore morale and instill a disciplined approach to planning and maintaining technology at PacifiCorp when he took over as CIO in August 1997.
"We created 74 service levels that the customers agreed to, ranging from network uptimes to response times to problems," Meier says. "These are measured on a monthly basis and posted on the company's intranet site, along with the department's goals and scores ranging from acceptable to excellent and showing clearly which targets were hit or missed."
PacifiCorp, which recently merged with ScottishPower, has 200 IT employees in Portland, and 100 more in Salt Lake City. Divisions within PacifiCorp's IT department earn incentive bonuses if they consistently meet 90 percent of their goals. Meier also instituted a scorecard system to measure costs, staffing and service levels on an ongoing basis.
When a department isn't hitting its numbers, it sets off an internal alarm, and Meier and his staff work to resolve the problem. Sometimes it's a budget issue, such as network availability due to outdated infrastructure in a power plant.
"The only way to fix that is to upgrade the infrastructure," Meier notes.
Sometimes it's a vendor, he says. "But usually it's a process issue we can work through."
In addition to the bonuses tied to the performance measures, there's peer pressure, because the results are published for everyone to see. "If we missed six of 100 service levels, but four of them were in telecom, those guys wouldn't like it at all."
Finally, PacifiCorp measures projects in three ways: costs, schedules, and meeting requirements. "The first two are objective and easily measured," Meier says. "The third is more subjective and requires user feedback.
"These measures are good for defense," Meier says. "Another exec might say, 'Gee we have a lot of desktop personnel.' But I can show them a measure that says our cost per workstation is [US]$2,700 per year, whereas the industry average is something like $5,000. As long as the measures have some validity, that's a pretty strong argument."
Ironically, it's when things are going well in IT that he has to watch out, Meier says. "That's when we're scrutinized closer. When they say, 'IT performance is good, but maybe we're spending too much,' that's a difficult conversation to have, because it's a slippery slope from good to lousy."
(David Raths (firstname.lastname@example.org) is a free-lance writer based in Kailua, Hawaii.)