Of the three exploits chosen for this test (the Cisco malformed SNMP vulnerability and the SQL Slammer and Witty worms), the SNMP one gave our IPS vendors the most trouble.

The SNMP exploit is a simple one: Many versions of Cisco IOS 12.0 to 12.3 will reboot when a valid SNMP message, such as a Get or Set, is sent to a destination port expecting an unsolicited message, such as a Trap.

For example, a Get query normally goes to User Datagram Protocol (UDP) Destination Port 161. If an attacker instead sends a perfectly well-formed SNMP query to Port 162 (the Trap port), vulnerable versions of IOS will reboot. Because reboot time on some devices takes minutes, an attack even at a very low rate could keep a vulnerable router, switch or access point off the network.

Cisco announced and patched the vulnerability in May 2004. Considering the exploit's age and high profile, we expected all IPSs to detect and block it without incident. To make sure our SpirentThreatEx vulnerability-assessment tool was firing a valid attack, we tracked down a vulnerable version of IOS, loaded it on a Cisco router and crashed it repeatedly.

Only the TippingPoint 5000E IPS spotted the exploit in initial testing. The other five IPSs tested missed it. In several cases, vendors asked us for a traffic capture so they could write a signature.

Even the 5000E missed one highly improbable variation of the exploit during initial testing. IOS listens for unsolicited messages on UDP Port 162 and on one random port in the range of 49152 to 59152. The TippingPoint device blocks solicited messages sent to either port.

If the randomly chosen port is in use, however, IOS instead listens on a port in the range of 59153 to 65535. Initially, the TippingPoint device did not spot exploits offered to a port in this second high-numbered range. Note that the probability of this vulnerability occurring is extremely low.

TippingPoint wrote a new signature to cover this remote possibility, but it had the effect of forwarding a small number of Cisco exploits when offered concurrently with benign TCP traffic -- even when offered to Port 162 (an exploit that TippingPoint previously blocked). TippingPoint gave us a second up-dated signature, but the problem persisted. TippingPoint has since issued a fix for this issue.

As we started heating up the technical-support lines for all the other vendors, the patched signatures showed up fast and furious. Some vendors rapidly developed a comprehensive signature that blocked any form of the exploit. Other vendors, reacting to our test, rushed the job and wrote signatures that were so vague as to virtually guarantee false positives and negatives.

What was interesting to us was how poorly the signature-writing process worked. Demarc Threat Protection Solutions delivered a signature for its Snort-based system, and to its credit, also contributed the signature to the open source signature repository at the Bleeding Edge Snort Web site. Demarc's first attempt covered only Port 162, but it quickly added signatures to cover exploits offered to high-number ports. Demarc did a great job, even though it took them a few tries to get it right.