There's an adage that says change is the only constant. Still, every IT executive knows constant change wreaks havoc on a complex IT environment. In fact, IDC and Gartner report that 70 percent to 80 percent of IT-related problems are directly attributable to changes made to the environment.
Mark Etherington, global head of distributed computing at JPMorgan Chase, New York, sees it firsthand. His firm makes thousands of changes to its 30,000-server environment each month. Like many financial firms, JPMorgan sometimes imposes change freezes at the end of the month or quarter to ensure optimal systems availability.
"When we have a change freeze, we see a reasonably dramatic drop-off of issues in the environment," Etherington says. "So we're faced with a conundrum. Do we prevent change to increase availability, or do we work out how to manage large numbers of changes better? The way to attack this problem is to make change a safer practice."
The problem is compounded as firms move to New Data Center technologies, such as virtualization. "As you build a more complex infrastructure to support things like server virtualization, you may think it's OK to be sloppy, since virtualization guarantees the service availability to some degree," says Richard Potocki, department manager of IT operations at Erie Insurance. No one would notice if 25 out of thousands of servers fail because virtualization would cover for them. "But . . . an environment that allows me to be that sloppy has to be very complex. To manage that complexity, to make sure it works properly, you need to have really good change management," says Potocki, who has automated change management across his 285 servers.
Good change management relies on automation, specifically automating the change-management process while following the best practices laid out within the ITIL, users say. Strict IT business processes implemented via automation can increase the success rate of change, thereby reducing the number of changes necessary, eventually resulting in increased service levels across the board.
But getting to that point isn't easy. Many tools provide some automation but not of the entire change-management process. Ultimately, end-to-end change-management capabilities should come from larger firms, such as BMC Software, CA, HP, IBM and Symantec. Each of these companies, for example, is integrating the appropriate technology, often gained through acquisition, into their product lines.
Before choosing a change-management product, get a handle on current processes to ensure that they are as efficient, manageable and auditable as possible, users say. "You have to understand what you want and put it in the right context. Otherwise, you tend to lose focus," says Stephen Ashton, London CIO at Dresdner Kleinwort Wasserstein, a global investment bank that runs 10 data centers.
ITIL comes in by spelling out best practices for six main IT business processes - configuration, incident, problem, change, service/help desk and release management - aimed at providing operational efficiencies. A truly automated change-management process includes elements of each of these six. For example, you can't make a change without first addressing how it would affect the environment, and you can't do that unless you know what elements are in your environment and their dependencies. This is configuration management, for which ITIL specifies the use of a configuration-management database (CMDB).
Similarly, many changes are designed to fix problems in the environment, so processes and tools that tie into incident, problem management or help desk systems are key. And once a change has been approved, tying it to an automated release-management system ensures that its rollout is controlled, tested and auditable. Each ITIL best practice provides key functionality and accountability to the overall change-management system, which is paramount for organizations facing compliance issues.
"If you look at the procedures investment banks have gone through with Sarbanes-Oxley compliance, you find things where you thought you had controls, but you haven't got evidence. We've been working on the levels of application, general and entity-level controls, and the ITIL framework is a good response, especially in change management," Ashton says.
Users say automating change can't work without a CMDB, because automation requires a good map of your environment. In a sense, the CMDB is a huge database that keeps track of every configuration item - be it hardware, such as a router, or software, such as an application release - in an environment. It also maps the dependencies among these items. So, if an application running on one server depends on a database on another server and uses a security appliance running elsewhere, then the CMDB shows all of those connections.
The problem has been getting that level of information in a single view and keeping it current. Today, there are several CMDB wares that provide automated discovery of configuration items and automated mapping of application dependency. These include products from BMC, CA, Cendura, HP, IBM, nLayers, Mercury Interactive, Symantec and Tideway Systems.