At SunTrust, a major bank with US$182 billion in assets, network executives had the blues. Although mergers and acquisitions had boosted the bank's branch and ATM presence across the Southeast, they had created a network-management morass.
By late 2004, the bank had thousands of branches and ATMs, which meant an increasingly heterogeneous collection of legacy switches and routers. The network was getting more difficult to monitor and maintain. Tedious, manual processes for making configuration changes were bumping up against the need to whittle the change window from eight hours a week down to four. And confidence in the bank's overall compliance stance was withering. "Answering simple questions, like whether baseline configurations were still valid, became increasingly difficult," says Tim Baker, manager of network-management and support at SunTrust, in Atlanta.
SunTrust's network managers considered hiring three staff members to fight fires while tenured team members devoted time to getting a handle on the environment. Such an approach would have cost US$300,000 in salaries and training alone, Baker says. Automating network-management functions seemed a likely alternative.
Last year, after conducting six-week, in-depth pilots of several tool sets, the team rolled out a proactive network- and configuration-management system that has eliminated faulty configurations across more than 6,700 devices, increased service levels and streamlined the compliance process. This network-management initiative earns SunTrust a 2007 Enterprise All-Star Award.
Automation is key
The bank automated network management using Opsware's Network Automation System, SolarWinds' Orion Network Monitoring, Fluke Networks' VisualUptime Select Application Integrity Analysis, as well as Opnet's Ace, NetDoctor and Flow Analysis trending tools.
With the Opsware tool, SunTrust sets baseline configurations for every network device and automatically checks those configurations every 24 hours. The tool flags violations -- for example, an unauthorized IP address with access to a critical server -- in its daily report.
Within five hours of bringing up the Network Automation System, the tool flagged more than 8,000 configuration-related problems. These ranged from high-priority items, such as improperly configured settings, to low-priority ones, such as typos and missing descriptions in config files. Still, within four months of the initial deployment, the bank had cleaned up all of them, Baker says. "Opsware really tightened things up," he says, noting that it helped reduce the number of exceptions from its auditor from five in 2006 to zero during the first six months of 2007.
This is because Opsware's ability to execute custom-built scripts streamlines the process. "It may flag 100 devices with the same problem, but you don't have to go touch every one of them," he says. "You just build a script to go out on a scheduled basis, and clean this stuff up automatically."
Once the bank had good insight into its overall network environment, it used the other tools to ensure optimum network performance. "SolarWinds' Orion tells us when we have a utilization issue or a potential problem that somebody will see at a branch location, where traffic is starting to slow," Baker says. "It has thresholds, and when it hits certain warning levels, we know we need to take a closer look."Â
The team then uses the Fluke tools to drill down to see whether the problem is the result of network bandwidth constraints or other factors, such as server or application-performance problems. "They'll tell us when a server is doing something strange -- they help us focus on the right place," he says.
Then, using the Opnet tools, the team pulls out more detail from the server or client -- finding and correcting errors before users notice a problem.
Scaling is a breeze
The new system has more than paid for itself, Baker says. SunTrust has realized an ROI of US$100,000 on its initial US$440,000 investment (for licensing, maintenance, installation and training) in the first year, in addition to the US$300,000 it saved by not hiring new staff. It also continues to reap efficiencies, a fact underscored when the bank brought up a new data center this April in just three hours, vs. the four days it would have taken previously, Baker says. "Click, click, click, and it was all over with."
All this makes the bank's growth less burdensome. "With these tools in place, we can easily double in size," Baker says.
The automation also reduces grunt work, letting IT staff focus on providing the best network services. In his 25 years with SunTrust IT, Baker says, "I've never seen any other deployment operate at such a high level, from functionality, efficiency and utilization standpoints."
Cummings is a freelance writer. She can be reached at firstname.lastname@example.org.