Overcoming Earth, Wind and Water To Keep Data Safe

FRAMINGHAM (03/06/2000) - Word came early in the evening on May 3, 1999: An F5-level tornado - the deadliest kind, with winds that exceeded 260 miles per hour - had hit Oklahoma City. Kevin McDonald, director of information services at Tontitown, Ark.-based PAM Transportation Services Inc., feared the worst.

Oklahoma City housed PAM's truck terminal and the dispatch center for the company's subsidiary, Choctaw Express Inc. McDonald had to keep that center up and running.

So he sent a damage-assessment team from Arkansas. The team arrived to find Oklahoma City a mass of debris and devastation. Although the twister barely kissed the dispatch center grounds, coming only within 200 yards of the metal-frame building, it had wreaked havoc.

Most of the contents, including some PCs, had been sucked out of the building.

A diesel truck was hurled a half-mile away. Trailers exploded from the air-pressure changes. Windows were blown out of the office, and rain had drenched much of the electronic equipment. Fortunately, no employees were hurt:

They had waited out the storm huddled in the long, narrow, 4-ft.-deep grease pits used to service trucks.

The team wrapped up the communication equipment to protect it from rain.

McDonald called SunGard Recovery Services Inc. in Wayne, Pa., and formally declared a disaster under the terms of PAM's service agreement.

SunGard contacted its Metro Recovery unit in Atlanta and sent a truck loaded with basic equipment configurations previously specified by PAM on the nearly 900-mile journey to Oklahoma City.

PAM called Little Rock, Ark.-based Alltel Corp. and purchased a duplicate phone system. A local interconnect company shipped it to Oklahoma City. In the meantime, incoming phone calls were routed to PAM headquarters.

When the maintenance director used a generator to restore power, McDonald discovered he was luckier than he originally thought: The office's frame-relay link, router and phone system still worked. But that didn't mean everything was back to normal. "The building was unusable," McDonald says. "There was no way anyone could work in there."

SunGard's Metro Recovery people arrived around 4 p.m. the day after the tornado, and "they basically picked up everything," McDonald says.

By 6 p.m., a bit more than 24 hours after everything was blown to pieces, the system was fully restored, and the dispatch center was back online.

Except for some initial confusion, McDonald says PAM's workflow was never seriously disrupted. That's largely because within an hour of the strike, PAM's in-vehicle satellite messaging and tracking system notified drivers that it would handle dispatching while Oklahoma City was down.

One factor, McDonald says, must be added into the disaster preparedness equation: Don't overlook the human element. When visualizing recovery scenarios, realize that employees may have overwhelming personal obligations to help family and friends during catastrophes, and that will limit the ability to staff a backup site internally.

Sharon Savings Bank wasn't as lucky last September. One of its bank buildings in Darby, Pa., sits next to a creek that was flooded by Hurricane Floyd's torrential rains. "It kind of took us by surprise," recalls network administrator Shirley Martin. "We didn't have much time to get outside ourselves."

The next day, workers found 8 feet of water in the building. Twenty-two PCs were smashed to the floor by raging floodwaters. A nearby administrative building was also out of commission, so workers in Martin's building had to set up shop in a nearby mortgage office.

The most critical data was safe: The bank's main database was kept off-site at an Electronic Data Systems Corp. division in Florida. But important documents, policies and account balances created in Microsoft Office and specialized applications were on the hard drives of the lost PCs.

Paper copies and tape backups (the latter stored in a bank vault) weren't viable restoration sources, so Martin asked her local maintenance contractor to remove four soggy hard drives and gauge the odds of data recovery. The flood had deposited caustic substances on the drives, so the consultant recommended sending them to Ontrack Data International, where technicians in "clean rooms" could safely remove the drive's magnetic platters and use special instruments to read the remaining data.

Over the next two weeks, Ontrack shipped back CD-ROMs containing nearly all the original data. "I'd say it saved us about three or four months worth of overtime work," Martin says.

Join the newsletter!

Error: Please check your email address.

More about AlltelChoctaw ExpressElectronic Data SystemsMicrosoftOntrack Data InternationalPAM Transportation ServicesSUNGARDTransportation

Show Comments