We were pushing for a speedy move to the supposedly more secure Windows Server 2003 -- until we ran into the vulnerability in remote procedure call (RPC) services that use the Distributed Component Object Model. Every version of Windows, including Server 2003, is vulnerable to this latest buffer-overflow flaw. So we're rethinking our plans.
Microsoft Corp.'s announcement and patch were swiftly followed by the release of exploit code, so my team and I knew a worm was sure to follow. The patch involved a reboot, which increased the amount of change control needed for rollout. We did pretty well anyway: We had over 80 percent of our vulnerable machines patched by early August, when the Blaster worm, which exploits the RPC flaw, hit.
Luckily, even the most trivial of firewalls stops this worm. But there are enough Windows machines without firewalls out there that the worm infected much of the Internet. Our firewall was jammed with thousands of attempts to get in, but we were unaffected -- at first.
We knew that even a good firewall isn't enough to stop a worm, so we began to analyze other routes such an infection might take. We quickly determined that our highest risk lay with laptop users.
We sent an e-mail to all laptop users and warned them not to connect to the corporate network without checking with the security team first. We also got an updated virus definition from our security software vendor, Santa Clara, Calif.-based Network Associates Inc., and began rolling that out. Fortunately, McAfee VirusScan doesn't require a system reboot on most installs. We could roll it out with minimal disruption and plan a more convenient time to install the Microsoft patch.
Our laptop users heeded our warning, and soon we were checking laptop firewall configurations and antivirus software signatures.
At this point, we felt proud of ourselves. We started thinking how foolish a big company like ours would be to allow itself to be hit by the Blaster worm. An easy patch, plenty of warning and a poorly written worm that could be blocked by the simplest of firewalls. How could you fail to stop it?
A few days later, we had our answer, as desktop after desktop lit up with virus-alert warnings for the Blaster worm and our intrusion-detection systems (IDS) went wild. Someone on the inside was spreading the worm.
We used our IDS to quickly isolate the source of the problem to one laptop user. While my colleagues frantically called the user to tell him to disconnect from the network, I literally ran down to his office. I was flushed, angry and out of breath from running across the building. I didn't bother with our normal incident process: Once I confirmed that I had the right machine, I confiscated the laptop without explanation and left. I felt so betrayed by the user that I didn't know what I'd say if I remained in his office.
Once I got back to my group, we had a few nervous, quiet moments monitoring the IDS. Fortunately, it had returned to green status with the removal of the laptop. A few more antivirus alarms came through, but these turned out to be delayed warnings.
We had caught the problem in time. No other machines had been infected, although we came frighteningly close to a meltdown. We'd been protected by the antivirus software on machines that didn't have the patch, and luckily the infected laptop hadn't tried to contact any machines that lacked the patch or didn't have an up-to-date antivirius signature.
We then analyzed the infected laptop. The antivirus signature was months old, and the firewall had been set to a wide-open, trusting configuration that had allowed the worm to break in.
After I controlled my temper, I returned to discuss things with the end user. He couldn't explain how the firewall had been changed, but he did admit that he had wanted to copy some files off his laptop before he gave us the machine to be checked, so he had ignored our advice and connected it directly to the network before letting us check it. This had allowed the worm to start scanning internally.
I'll have to put more effort into convincing laptop users to trust us, and I'll have to find ways to enforce the behaviors that we want. But it's clear that our pride in our defenses was misplaced. We need to patch quickly and have up-to-date antivirus software on every device in order to stand any chance of surviving the coming waves of worms.