Advanced preparation

What enabled Lagasse to survive Katrina was a practical plan forged by the trial and error from the previous hurricanes. "I can learn if I'm hit over the head by things and that's what happened in this case," Lancaster said. When Hurricanes Isadore and Lilli hit in 2002, the company's disaster plan included assumptions that didn't pan out. Things were better by the time Ivan hit in 2004, when the company was forced to declare a disaster and run its operations from the Chicago backup site for five days.

"There's no better test than actually doing it and running your business that way," Lancaster said of the Ivan experience. "This wasn't testing. This was real live fire."

One of the lessons learned was the importance of coming up with the tiering strategy that dictates the order in which applications are brought back online following a disaster. Lancaster sought to come up with tiers that are easy to understand and communicate to the business side.

Tier 1 applications are those specifically required to generate revenue. For Lagasse, that means the ability to take, pick and ship orders. The goal is that such applications suffer no more than 15 minutes of data loss and be recovered within six hours. "That was deemed acceptable by the business, especially considering we're maintaining a low cost profile," he said, noting the organization's IT budget was just 0.8 per cent of revenue.

These applications should also be recoverable through semi-automated means and without assuming that specific, highly knowledgeable people are available. The use of scripts and detailed documentation meant personnel with good IT knowledge would be able to recover the resources, but it didn't necessarily require the same people who work with them every day, he noted. Applications such as the company ERP system were consistently replicated to the Chicago site via a 3Mbps frame relay link.

Tier 2 applications are those that have to do with the customer experience. Essentially, that means anything that customers would notice if it were down, such as online order entry and various reporting applications. For these applications, the company is willing to lose as much as 24 hours of data and live with a recovery objective of three days. Less automation is involved in recovering these resources and it can be difficult without specific IT staffers.

At Tier 3 are computing resources used only internally, that won't be noticed by anyone outside the company. "They'd only hurt us," Lancaster noted. The IT group makes no specific commitment as to when it will recover Tier 3 applications, he said.

"Spending a lot of money and adding a lot of complexity to become very good at recovering Tier 3 applications really wasn't very value added," Lancaster said. "We'd rather hit the Tier 1 and Tier 2 [applications] 100 per cent and worry about the Tier 3 when the time comes."

Post-Katrina, Lancaster said some adjustments were in order in terms of how applications were classified. Financial systems, for example, fit the Tier 3 definition. But Katrina hit in late August, and September is the last month of the quarter. By September 8, Lancaster was hearing from the CFO about Securities Exchange Commission regulations.

Likewise, e-mail was originally classified as a Tier 3 application. But in the wake of Katrina, "We found e-mail to be about the most valuable communication tool we had at our disposal," Lancaster said. "It very quickly escalated to Tier 1."

Another key to Lagasse's successful disaster recovery plan was keeping its application architecture simple. Whenever possible, his group strives to be involved in defining the solution to a business need, rather than having solutions forced on it. "When an application gets forced on you, it often has architectural principles that are not aligned with what you do, and so you're not very good at [supporting the application]. It just makes things a lot harder," he said.