Last week, my team and I discovered a vulnerability in the Cisco Systems Inc. equipment we use in our global network. There are 253 possible IP-based protocols in IP Version 4, and the majority of Cisco routers and switches have a serious problem with four of them. The flaw leaves unpatched equipment open to denial-of-service attacks.
Once the Cisco device receives a certain number of IP packets of Type 53, 55, 77 or 103, it stops functioning. If a switch or router doesn't know what to do with a given packet, it just leaves it in the queue until the queue fills up and the device stops working.
The first reports of this vulnerability made it clear that the packets had to be targeted at the router being attacked in order to succeed. I immediately thought we would be fine, since our core routers have access-control lists (ACL). We set these up to operate like a minifirewall that can allow and deny various kinds of traffic.
To protect our routers, we set a rule that routers accept specific traffic types coming only from our internal management machines. We don't bother listing every kind of bad data. Instead, we drop everything except the handful of things we need. So our routers drop those four vulnerable protocols without processing them, along with every other IP-based protocol except TCP and the User Datagram Protocol.
This meant we didn't have to do anything. Or did it? We checked our internal routers to make sure the right protections were in place and then performed the same check on our Internet-facing routers. Our firewall drops the four protocols mentioned earlier, so it would be difficult for someone to attack our internal routers. However, the external routers that connect to multiple Internet service providers have to be outside the firewall, and so they might accept those protocols if they were misconfigured.
I checked our external-facing routers from a remote provider's site, connecting to each and scanning to see on which protocols and ports the routers were listening, and I was very surprised when one answered on Telnet.
We use Telnet to manage some of our routers because not all versions of Cisco's Internetworking Operating System (IOS) support Secure Shell, our preferred encryption method. But Telnet wasn't supposed to accept connections from outside our company. The router's ACL should have limited connections to only those from authorized devices with addresses internal to our network.
As it turned out, the ACL had been applied correctly, and other traffic was being dropped as designed. Then I noticed that the IP address in the rule didn't match the IP address of the router I was examining. It belonged to another Internet-facing router. A network administrator must have cut and pasted the rule set for the router's ACL without editing the IP address.
After we corrected the IP addresses in the ACL, we thought we could rest easy: No attacker could get any flawed data to our machines.
I have to despise attackers. It's a professional requirement of the security field to hate those who make our lives difficult. But once in a while, I have to give the brightest ones a bit of respect.
Some clever person figured out that you might not have to send traffic to a router to get it to process the data. Every packet on the Internet has a Time To Live (TTL) counter setting, and every time a router handles a packet, its number decreases by 1. This keeps packets from circulating forever.
If the TTL reaches 0 while passing through a router, then that router must process the packet to decide if someone needs to be sent a warning that the packet didn't make it. I received an e-mail security alert from a trusted source that said if you arrange the TTL of Packet Types 53, 55, 77 or 103 so that they reach 0 just as they hit a Cisco router like ours, that router will process the packets despite the ACL settings. The packets won't match the ACL, as they aren't destined for that router but for addresses behind it. If the router processes packets with these four IP-based protocols, then the packets will get stuck, and the router will fill up and stop.
That's clever but annoying, because it meant we would've had to make sure that we had deployed not only the right ACLs but also the new versions of IOS to fix the problem. Each time we thought we had this problem under control, it popped up again. We can test a new release of IOS, but it takes time and is risky to deploy, whereas ACLs are well understood and low risk.
Then, as we were rushing about with our testing, Cisco contacted us to say that this risk doesn't exist. It said the routers discard the TTL packets without problems. Were the hackers wrong? I have to trust that Cisco knows best what its equipment can do.
Once the new version of IOS is out of testing and deployed, we'll be safe. Until then, we'll closely monitor how well the ACLs are protecting us.
The strangest thing about this whole issue has been that a large number of our customers have asked what we're doing about it. I don't understand this. I would never ask another company what it was doing, since the answer wouldn't cause me to do anything differently.
I also don't have the resources to ask every supplier what it's doing about such issues. Rather than try to have enough people free to collate all that information, I'll just protect myself from the possible attack, be safe and not worry about what others do.