Thinking about disaster recovery, a year after Hurricane Katrina, and on the fifth anniversary of 9/11, it's a good idea to look at contingency planning, especially for the organizations that would need to operate during a disaster. This is a different issue than transferring operations to a remote site. There are various issues in operating in a disaster environment, usually starting with electrical power but also extending to employee concerns and the need for resupply.
When demand gets beyond the capacity of uninterruptible power supply batteries, even with those as big as the 48-hour supplies of carrier-grade telephone central offices, there will be a need to generate power. Diesel generators are the most common, in part because diesel fuel is much safer than gasoline. The engines also may need other resources besides fuel.
Diesel fuel also has a definite storage life, and diesel engines powering generators should be run on a regular basis. Regular test runs help recognize pending or actual problems, keep the engines lubricated and help with turnover of the fuel. Diesel fuel tanks need maintenance, and possibly periodic testing of the fuel. Water can collect in the tanks from condensation of air. Obviously, water is not a good thing to have in the engine, but there can be worse consequences.
It is not a joke to say that evil germs, specifically of the Methylobacter genus, turn fuel into slime. These bacteria can grow at the interface between water and fuel and form mats of cells that can then break up and clog fuel filters, injectors, etc.
One way to avoid water condensation and also reduce fire risk is to have the exhaust of the engine piped back to the fuel tank so that it replaces air. With careful design, the exhaust arriving at the tank will be cool enough for fire safety, but it may be set up to contribute to the warming of the fuel. In cold weather, diesel fuel thickens faster than gasoline, and fuel tank heaters will be needed in cold climates. Any additional heat source reduces the power needed for tank heaters.
Tank, generator and battery placement
Lessons about locations for fuel tanks came from both Sept. 11, 2001, and the hurricanes. Without touching the various conspiracy theories touring the Internet, it is clear that having the fuel tanks for the emergency generators on the roof of World Trade Center building 7, which housed New York's Emergency Operations Center, did not contribute to continuing operations when oil cascaded down but instead added to fires.
Other issues may keep the diesel generators from working. If the tanks simply have a spark-shielded vent to let air come in to fill the space created as fuel is used and the vent is covered with floodwaters, water will be sucked into the tank. Tanks with an exhaust return can be engineered to work underwater.
Remember that the diesel generators and their tanks can be a significant distance apart, as long as fuel can be pumped by the available pump motors, and that there is a proper method of getting air or exhaust into the tanks to avoid suction locks.
Large battery banks, with liquid acid electrolyte, often can't be placed above a certain level of the building because of the local fire code. The concern is acid raining onto fire fighters.
Components forgotten about
After 9/11, surviving carriers, Internet service providers and financial institutions continued to operate on their diesels. Typically, such installations had a week of fuel and a plan, coordinated with fire and police officials, to have resupplies driven to them. Many operators were shocked, however, to find the engines working poorly, and then stopping, after 24 to 48 hours.
They stopped because the massive dust contamination clogged their inexpensive air filters, and they had no replacements. Luckily, these were small and could be delivered by police cars. At subsequent network operators' meetings, it was found that 9/11 was not unique in putting significant dust in the air. This had happened after the Mount St. Helens explosion, but also after large forest fires in the northwestern U.S. and western Canada.
Another problem is that while generators are too large to steal, starter batteries do have a certain street value. Wise operators keep an extra battery inside and on a trickle charger.
-- Test diesel generators about once per week. Periodically, test the automatic power fail-over and have the fuel tested for contamination.
-- Fuel tanks don't need to be next to the generator engines. There are potential problems if they are too high or too low. Tanks too low can be designed to work underwater, but it won't be easy to refill them. Middle levels of parking garages can be good locations.
-- Have contracts in place for fuel refills. Store spare fuel and air filters. Keep a starting battery inside and charged.
-- Be sure your facility, especially if it is windowless, dedicates some power to necessary heating, ventilating and air conditioning.
-- If your water supplier loses electrical power, you may not have running water. Generous supplies of bottled water will make life easier, along with alcohol sanitizers and personal wipes. Rainwater or even floodwater can be used. If you use floodwater, put bleach in it simply to make it safer to handle.