Batteries don't like generators -- use line conditioners
Faulkner's next challenge came at 11.30pm when the Internet went down. A quick check showed what had happened. Although the old server room was getting power, it had no line conditioners, so the generator was connected to the APC battery packs. But the batteries were not charging, and when they died, power to the Web, e-mail and DNS servers went with them.
Faulkner put a line conditioner on the power supply and hooked it directly to the critical servers, which gave them power, but when he tried to restore using Windows tools, he discovered that the crash had sent the read/write heads wandering across the disk of the DNS/Web server, damaging the partitioning. He needed to restore.
Backup, backup, backup -- but not to tape
Fortunately at about that time, as part of its emergency preparedness, the council had moved from tape backup to the Acronis True Image Server. The old tape system required 72 hours for a bare metal restore. "Fortunately, the servers went down after Acronis finished its automatic 11pm backup. I put the backup CD into the drive, initiated a restore and held my breath. Eight minutes later it said 'Finished' and rebooted the system. And the server came up as if nothing had happened." He has been an Acronis fan ever since.
Always keep the primary DNS server separate
"Never consolidate the DNS and Web servers on a single machine," Faulkner says. "If you do, you are asking for trouble."
Protect your emergency generator
The generator at the local hospital took a pipe through its radiator, putting it out of action. At the fire/rescue station the storm ripped the cover off the generator. It continued to function, but with the circuits soaked the power was very dirty. "We never did find that cover," Faulkner says.
Lightning strikes twice
Three weeks after Charlie, Hurricane Frances hit the region. The centre of the eye passed 10 miles northeast of the town. Again, there was little warning, and while this was a Category 2 storm, it also moved much more slowly, and the area was still recovering. The emergency management centre was down to four of its 25 workstations. The rest had fried their power supplies on the dirty power from the old generator. The school system loaned them 15 laptop computers, but they arrived six hours before Frances, stripped to bare metal.
Faulkner spent four hours building the image on the first laptop. But then he simply used Acronis True Image Workstation to clone that image onto the others. In 22 minutes he had all 15 up and running.
So by the time Hurricane Jeanne arrived a few weeks later, the area had one of the most experienced emergency management teams in the state. Jeanne was a Category 1, barely a hurricane in terms of wind, but it was a slow-moving rainmaker. "It was never made official, but we estimated we had a 500-year flood event. We had water where we never had seen water before."
Today the emergency team is all on laptops, which have several advantages. First, they have their own, built-in line conditioners. "A laptop can convert any power that comes at it into the right voltage."
Second, if the power goes out they keep working. Third, you can pack and run if you have to. "If we have to abandon our location, we can grab an access point, pick up our laptops and relocate," Faulkner says. His laptop is configured to function as a Web, DNS, & e-mail server if necessary, so they can set up anywhere and be back online in minutes. And with portable wireless access points and the backpack satellite transceiver, they can carry their connectivity with them.
Today, one of Faulkner's hobbies is running "what if" disaster scenarios. And since he knows he will never think of everything, he asks people in other departments and outside the council for their thoughts. Ultimately, flexibility is the key. Things like the Acronis quick restore and cloning abilities and maximum portability give the emergency team the ability to react to multiple contingencies.
"Statistically, we know that an area that is hit by a major hurricane is likely to get hit again within two years," he says. Faulkener's area was hit by three hurricanes in a row two years ago. This year, he is determined to be as prepared as possible.