You may have a disaster recovery plan in place, but are you prepared for a true catastrophe?
If you're fortunate, you don't have personal experience dealing with tragedies like the September 11 terrorist attacks or Hurricane Katrina. But one of the best ways to prepare for such disasters is to learn from those who survived them.
Tulane University, Hard Rock Casino & Hotel, Hancock Bank and others learned some tough lessons during the decade's deadliest disasters. And now they're sharing their hard-won knowledge.
1. Plan a backup site in advance
THEN: After Hurricane Katrina hit, Tulane University's IT team was able to recover its backup tapes, but the New Orleans data center was without power, and no backup site had been prearranged.
NOW: Signed on as a SunGard customer, the university is entitled to a mobile data center that could be used for local processing. And backup tapes are now sent to Baton Rouge three times a week.
2. Take control of your telecommunications
THEN: Like many other companies, insurance firm FA Richard & Associates hadn't expected the local telecommunications failure triggered by Hurricane Katrina.
NOW: FARA signed up with various mobile phone providers using multiple area codes. The firm can also now reroute its own 800-number services in emergencies, via the Web.
3. Be ready for extended power outages
THEN: When Hurricane Katrina devastated the Louisiana coast, a lack of batteries and power had Hard Rock Hotel & Casino employees relying solely on text messaging to communicate.
NOW: As part of its revamped disaster recovery plan, IT employees carry car chargers for their mobile phones. There's a new personnel plan for disasters, too.
4. Choose a "low-threat" data center location
THEN: During Hurricane Katrina, Hancock Bank's Gulfport data center, just one half mile from the Gulf of Mexico, was devastated.
NOW: The bank's new US$16 million data center is still in Gulfport, but the hardened, lights-out facility is located farther inland on the highest point in the area. It can withstand 200 mph winds.
5. Speed up server file replication
THEN: When Katrina struck, Hancock Bank's virtual server files could be quickly set up on hardware in a backup data center. But getting them there and loading from tape took 36 hours.
NOW: A new system reduces the boot recovery process to about 45 minutes.
6. Layer on communication methods
THEN: As communication channels began to flicker back on in Katrina's aftermath, companies throughout the region, including Marriott, found that different telecommunications components were fading in and out.
NOW: Marriott's recovery teams learned that the best way to keep communication channels open among employees was to use a mix of mobile phones and BlackBerry devices with different carriers. Having discovered that pin-to-pin communication and texting were the most reliable solutions, the disaster preparedness team quickly educated employees on the features and made it part of the disaster recovery plan.
7. Create a mirrored infrastructure
THEN: In 2004, when Hurricane Gaston stalled over for hours, Estes Express Lines' first-floor data center was awash in four feet of water. Company executives watched helplessly as 185 terminals used to direct the operations of more than 20,000 tractor trailers just died. All told, Estes had US$16 million in hardware losses.
NOW: The company pieced together a new infrastructure, complete with software that allows data to be whisked off-site immediately. That new architecture is mirrored in a hurricaneproof backup site nestled in sunny and dry Arizona.