Beth Everett's brushes with history have left her with better insights into worst-case scenarios.
Everett worked at Novartis in 2001 and saw the company's global connectivity cut off when the network lines under the World Trade Centre were destroyed.
Then she worked as CIO at Organon International, which shared space in commercial district with a large financial services institution high on the list of potential terrorist targets. What would happen, she remembers thinking, if officials shut down the area and employees couldn't get to work?
And now, as a consultant at Network Inference, in California, she wonders how a potential hazardous materials situation -- a chemical spill or attack -- might affect IT. Worries like these would have been unimaginable even five years ago. But world events have changed the way many CIOs think about their disaster recovery plans.
"There are definitely new threats out there," says Steven Agnoli, CIO at law firm Kirkpatrick & Lockhart Nicholson Graham.
Some CIOs are imagining potential disasters that go well beyond the everyday hiccups that can disrupt applications and networks. Others, recognizing how integral IT is to business today, are focusing on the need to recover instantaneously from any unforeseen event. Many are trying to do both. But CIOs agree that disaster recovery planning has taken on an immediacy that didn't exist in the 90s.
And they expect the threats to get worse. "The things you think about are, What will the virus/hacker people be able to do 10 years from now? What do I need to do to keep my capabilities ahead of the game?" says Rob Reeg, senior vice president of global operations at MasterCard International.
"We need to be prepared for the next level of alert," says Joseph Daluz, vice president and CIO at Computer Horizons.
Old worries and new
When it comes to disaster recovery, the concerns are diverse. CIOs say they still worry about the traditional problems, from those manual errors and little snafus that can crash a system to natural disasters like fire and flood. But they've also added new concerns that range from catastrophic power loss and network attacks to employee sabotage and terrorist attacks.
Raj Sampath, chief technology officer at LoanCity, a wholesale residential mortgage lender, has considered just about all those scenarios. He says his biggest fear is a hacker attack. "It's the unknown part -- I don't know how or when it's going to be," Sampath says.
He says a successful attack could not only corrupt his system but also compromise the personal data of the company's customers. That's why he has a firewall manager -- a combination of hardware and software that acts as the first point of contact for the external world, manages the security of the company's systems, protects servers from hackers and allows only certain specified transactions. Sampath also diligently keeps security software updated.
He worries about other scenarios, too, such as earthquakes knocking out his primary data centre and employees downloading infectious programs. So he sets up redundant systems, continually updates antivirus software, monitors employee computer use and uses technology that automates the backup and recovery process for Microsoft Exchange and SQL and Windows servers.
Other CIOs draw their new list of concerns from current events such as terrorist attacks and the blackouts. They ask, "What if someone sets off a dirty bomb? Or launches a bioterrorist attack? What happens if the country's ageing power grid fails?"
"It's a different world. There are so many more things to consider than the traditional fire, flood and theft," says Robert Rosen, a CIO in the US government and president of Share Inc, an IBM user group.
For example, as he toured a disaster recovery site last year, Rosen was impressed by its meticulous planning and features. Still, he was concerned when he heard a low-flying plane overhead, noting that a site's proximity to an airport -- even a small one -- means there's an increased risk of it being hit by a crashing aircraft.
"There's some risk there, even if the risk is pretty small. But it's one of the things you factor into your analysis," he says. "If I were with an organization that had to have total uptime no matter what, maybe I couldn't live with that risk."
But that doesn't mean CIOs have to prepare for every scenario they can imagine. Companies usually make their disaster recovery decisions based on cost and risk analysis, says David Palermo, vice president of marketing at SunGard Availability Services. They prioritize risks according to the likelihood of various scenarios and the effect each one might have. "At some point, you're out of money and you have to make your choice," Palermo says.
A key to risk analysis is that it's not always about full-blown system failures; even small problems can have significant consequences. "Disaster has taken on new meaning in this era of Sarbanes-Oxley and government regulation," says Mike Kahn, managing director of The Clipper Group, a technology acquisition consultancy. If regulators come knocking, they now expect companies to produce all the required data within hours -- not weeks, as they once did, he says.
But regulations aside, "data is now just so important to ongoing operations [that] if you lose data and it's real-time data, it could actually impact your business," Kahn says.
The best-prepared executives recognize that speed is essential in recovering from whatever disaster might come to pass, Kahn says. That's why there's a trend toward enabling technology users to restore their own documents, so a lawyer, for example, can retrieve a brief that took weeks to write but an instant to accidentally delete.
And in addition to regularly backing up to tape that's then stored off-site, companies are employing newer technologies to take snapshots -- every five minutes, or every hour, depending on the business -- to reduce the risk of potential loss, Kahn says.
But while the technology exists to ensure that a company doesn't experience a catastrophic loss of data even if its systems go down, experts say executives need to make disaster recovery a priority -- and fund it appropriately -- if they want to guarantee business continuity during almost any situation.
"There's no lack of products, and no lack of salespeople out there who want to sell them to you," Rosen says. "So it's not a technological issue any more. It's really a management issue and a will to do it. One of the problems is it's not cheap."
Robert Borr, CIO at a medical centre, has tried to protect his organization from just about any disaster that could take out his systems. In addition to strict backup procedures, he has agreements with hardware vendors to deliver new equipment within 24 hours if needed.
He pays extra for that but says, "If we can get computers delivered within 24 hours, we can be up and running seven or eight hours after that." Others have taken even more aggressive approaches to guarantee that they're prepared for anything. Agnoli says his law firm increasingly focuses on building in redundancies to ensure that if an office system is taken out or an e-mail server goes down, another one can quickly take over, averting a crisis while the problem is being fixed.
"The goal," Agnoli says, "is to avoid ever having to get into a disaster recovery situation if we can."