Working during a hurricane: 'It was a madhouse'
- 16 August, 2006 11:52
Two years ago, Hurricane Charlie hit rural Hardee County in central Florida with Category 4 power -- the same rating Hurricane Katrina had in 2005 when it hit New Orleans. In two hours starting at 5:15pm, Charlie levelled nearly everything standing -- virtually every tree, radio tower and light pole.
The telephone service in the county's centre is underground, but the storm pulled many of the cables out of the ground, caught in the roots of 200-year-old oaks. By the time it was finished, 75 percent of the buildings in the county were destroyed or severely damaged. "We are still pulling down homes destroyed by the storm two years later," says Don Faulkner, the sole IT staffer for the county's emergency management centre. "Weeks later, I would see concrete block walls fall. The storm had broken the mortar, and suddenly they would just sag and collapse. Today the tallest tree in the county is about 10 metres high."
Faulkner said the experience held a number of lessons:
Charlie arrived just as Hardee was moving its server system into a new emergency management building, actually its old jail, with two-foot-thick walls and windows protected by armored screens "that can take a hit from a 2-by-4 at 60 mph without denting", says Faulkner. But on August 13 2004 the move was not complete. The building had emergency power from a diesel generator, and the council's T1 connection had been moved to it along with the detached storage server.
"Our other servers were still in the old server room in the building next door, which had no emergency power. As an emergency measure, I ran 120 metres of 12 gauge extension cord from the generator to the old server room and shut down everything non-essential. I brought back backup tapes, disks, basically anything I could carry, and covered everything else in plastic in case the rain got in." As it turned out, that extension cord, lying on the ground between the buildings, survived when everything else failed. A good thing, because the county's DNS, Web and e-mail servers were in that room, and for four days after the storm the only communications the county had with the outside world was e-mail.
Hope for the best, but prepare for the worst
That morning Charlie seemed to be heading somewhere else, and the area was at Level 2 on the emergency management system. "At 11am we heard it had changed direction and was coming directly at us," he says. "Then it speeded up. So we only had a few hours to prepare. It was a madhouse."
At 5.15pm, wind speeds reached 45 mph with gusts to 60 mph, and the electric utility shut down power to protect its substations. At that wind speed, fire engines and ambulances cannot operate safely. But the storm was only starting. An hour later, the centre of the 10-mile diameter eye passed one mile east of the town.
"We were on the second floor of the emergency building, watching out the armoured windows," Faulkner says. "My parents' house, where my family was gathered, is two blocks away. I watched 100-year-old oaks topple over on top of it. Actually, that might have been good. They did some damage, but they held the roof on."
When the wind reached 120mph, it tore the weather station off the top of the emergency management building, so no one knows just how strong the wind got. They estimate 35mph gusts above the 145mph base speed.
"I watched one of the communications towers break off 15 metres up after being hit by flying debris," he says.
Two hours after it started, the wind was back down to 45mph and dropping. Charlie had passed. But it left utter devastation behind.
Plan multiple forms of redundant communications
When the storm passed, the are had no power except for emergency generators, no wireline telephone service, no surviving mobile towers. "Anything on a pole was gone," Faulkner says. "Even the satellite phones wouldn't work for some reason."
The emergency management director looked around and asked what his people could do. "I said, 'The Internet is still working.' So he sent an e-mail out asking, 'Is anybody out there?' His hands were shaking. When he received a response, there was cheering and a sigh of relief. For the next four days, the Internet was the area's only link with FEMA, the state emergency management office and the outside world in general.
Actually, the survival of the Internet connection was almost accidental. The council had an older T1 installation that was powered from its end rather than from the phone company, and it was hooked into the emergency generator. Without that, the area would have been totally cut off.
Today there's DSL, wireless satellite and a backpack satellite transceiver. And it has a new T1. "The installers wanted to power it from the phone system, but I insisted that it be powered from our end and hooked into the generator." And it has a new propane generator that can go from cold to full power in seconds.
Batteries don't like generators -- use line conditioners
Faulkner's next challenge came at 11.30pm when the Internet went down. A quick check showed what had happened. Although the old server room was getting power, it had no line conditioners, so the generator was connected to the APC battery packs. But the batteries were not charging, and when they died, power to the Web, e-mail and DNS servers went with them.
Faulkner put a line conditioner on the power supply and hooked it directly to the critical servers, which gave them power, but when he tried to restore using Windows tools, he discovered that the crash had sent the read/write heads wandering across the disk of the DNS/Web server, damaging the partitioning. He needed to restore.
Backup, backup, backup -- but not to tape
Fortunately at about that time, as part of its emergency preparedness, the council had moved from tape backup to the Acronis True Image Server. The old tape system required 72 hours for a bare metal restore. "Fortunately, the servers went down after Acronis finished its automatic 11pm backup. I put the backup CD into the drive, initiated a restore and held my breath. Eight minutes later it said 'Finished' and rebooted the system. And the server came up as if nothing had happened." He has been an Acronis fan ever since.
Always keep the primary DNS server separate
"Never consolidate the DNS and Web servers on a single machine," Faulkner says. "If you do, you are asking for trouble."
Protect your emergency generator
The generator at the local hospital took a pipe through its radiator, putting it out of action. At the fire/rescue station the storm ripped the cover off the generator. It continued to function, but with the circuits soaked the power was very dirty. "We never did find that cover," Faulkner says.
Lightning strikes twice
Three weeks after Charlie, Hurricane Frances hit the region. The centre of the eye passed 10 miles northeast of the town. Again, there was little warning, and while this was a Category 2 storm, it also moved much more slowly, and the area was still recovering. The emergency management centre was down to four of its 25 workstations. The rest had fried their power supplies on the dirty power from the old generator. The school system loaned them 15 laptop computers, but they arrived six hours before Frances, stripped to bare metal.
Faulkner spent four hours building the image on the first laptop. But then he simply used Acronis True Image Workstation to clone that image onto the others. In 22 minutes he had all 15 up and running.
So by the time Hurricane Jeanne arrived a few weeks later, the area had one of the most experienced emergency management teams in the state. Jeanne was a Category 1, barely a hurricane in terms of wind, but it was a slow-moving rainmaker. "It was never made official, but we estimated we had a 500-year flood event. We had water where we never had seen water before."
Today the emergency team is all on laptops, which have several advantages. First, they have their own, built-in line conditioners. "A laptop can convert any power that comes at it into the right voltage."
Second, if the power goes out they keep working. Third, you can pack and run if you have to. "If we have to abandon our location, we can grab an access point, pick up our laptops and relocate," Faulkner says. His laptop is configured to function as a Web, DNS, & e-mail server if necessary, so they can set up anywhere and be back online in minutes. And with portable wireless access points and the backpack satellite transceiver, they can carry their connectivity with them.
Today, one of Faulkner's hobbies is running "what if" disaster scenarios. And since he knows he will never think of everything, he asks people in other departments and outside the council for their thoughts. Ultimately, flexibility is the key. Things like the Acronis quick restore and cloning abilities and maximum portability give the emergency team the ability to react to multiple contingencies.
"Statistically, we know that an area that is hit by a major hurricane is likely to get hit again within two years," he says. Faulkener's area was hit by three hurricanes in a row two years ago. This year, he is determined to be as prepared as possible.