In the late 1990s I worked for a Midwestern manufacturer. We ran a very small IT shop for the size of our corporation (8 people for a company grossing a third of a billion dollars annually, across 10 locations), and all of us in IT were expected to wear many hats. I was a programmer, analyst and project manager when I wasn't network administrator and tech support. From our corporate office, I was directly responsible for five factories around the Great Lakes area, and I also helped out the MIS staff at our factory just down the road.

A year before, I had responded to a Friday plant disaster that left inventory and manufacturing untouched but destroyed the office space and all computers. Thanks to spare equipment and offsite backup tapes, our Monday morning shipments went out on time, saving the company hundreds of thousands of dollars. This event opened the purse strings a bit, and I was allowed to purchase a backup server to use in case one of the other factories had a similar disaster.

Like many companies at that time, we decided to jump on the ERP bandwagon. All the upper-level execs were involved, as we were going to avoid all of the mistakes that other companies were making, and have a cost-effective and pain-free implementation. I was not involved in the planning, being too low on the totem pole. For the next several months, the bigwigs confabbed while I set about improving our network infrastructure and documenting in-house applications. I couldn't work on any new projects, as we didn't know which ERP we would be getting and what would be made obsolete. At one point, the VP of Finance asked the CIO why I still worked there. To his credit, my boss said that although I was not going to be involved in the ERP planning, I would be instrumental in the installation and training. I, of course, brushed up the resume and prepared for the inevitable.

One Wednesday morning I was in my cube drinking my first cup of coffee, when I heard the unmistakable sound of the disaster server being removed from its home in the next cube. I stepped around the partition to find Bill, the head of MIS for the plant down the road, holding an armload of server components with a panicked look on his face. I asked him what was up.

Bill explained that there had been an extended power outage at the plant, and the UPS system had shut down the server in a controlled fashion. When the power returned, Bill turned everything back on, only to find that all the hard drives in the external RAID array were dead. The MIS staff tried everything they could think of but couldn't get the drives working again. They were going to swap in the disaster server.

Furthermore, the shutdown had occurred right in the middle of our lengthy nightly backups, so they were going to have to restore Monday night's tapes. Because the restore would have gone well into the evening, on Thursday morning the plant would be where they'd been Tuesday morning, with two days of productivity completely shot.

I asked Bill how the server had been powered up, and he said he'd just flipped the switch on the UPS. I recognized the problem immediately (having installed that server). This particular model of RAID array needed to be turned on and fully booted before the server was turned on. If the server's RAID card queried the RAID box before the box was ready, all of the drives would get marked as dead. The vendor had added a command line switch in a hot-fix that would resurrect the drives if this occurred.

I told Bill to put the spare server down and get on the phone right away to tell everyone not to touch anything. I jumped in my car and flew to the plant. Ten minutes later the server was up and running, with Tuesday's data intact and Wednesday's productivity still ahead.

Bill gave me some grief about undocumented switches, until I reached over to the bookcase next to the server and pulled the printed instructions out of the front of the RAID manual, where I'd put them when the vendor sent them to me.

By my estimate, I'd saved the company five times my annual salary in lost productivity that day. It was a really good day, too, because I received a job offer from a new employer that afternoon. When I handed in my letter of resignation to the CIO on Thursday, I learned that the VP of Finance had once again questioned my employment, on Wednesday morning while I was off saving the day. He'd suggested that the company could save money by letting me go for now, then hiring me back when the ERP project finally required technical people.

A year later, the ERP project was canceled. It never got beyond the planning stage.