It begins with a shiver, a vibration almost too faint to be sensed. My attention is pulled from the meeting I'm in by the security problem I know is occurring on our live network.

Do I have a strange power derived from the bite of a radioactive spider that allows me to sense the problem? No, I'm lucky enough to be one of the many security professionals bound by the electronic leash of a pager. I'm not always on the front line, but today, it's my turn to possibly have sleep and social life interrupted by the vibrating black box. At least this time, it pulls me from a tedious meeting.

The pager is linked to our security monitoring and alerting system and provides a summary of any incident detected. It has plenty to say today.

A glance at the messages indicates their urgency, so I make my excuses to leave the meeting and run back to my desk, warming up my team by cell phone as I go. En route, I receive yet another cluster of pages indicating that this isn't just an isolated event but something more worrisome.

In our office, the team has pulled up the full data flowing from our network and host intrusion-detection system. It doesn't look good. We see a cascade of thousands of attempts to log in to a critical host, using a variety of well-known accounts, such as root, guest, user and system. The speed of the attempts shows this is no spotty teenager in a bedroom typing attempt after attempt. This is a scripted attack.

The Source

The attack comes from a trusted business partner, and the server being attacked is deployed on its remote site. It isn't subtle, but few of the organizations I've worked with would detect it.

We see many attacks from the Internet, but we expect these and are well prepared to respond to them. This kind of attack brings with it the concern that our business partner has been compromised. Could it have a malicious employee? Has a hacker breached its network?

The server under attack is critical because it processes many financial transactions. Although none of the current log-in attempts will work, they are traditionally the precursor to more involved and skillful probes. Should the attacker move on to a more sophisticated attack, we'll be forced to choose between pulling the plug and suffering downtime and lost business, or letting the attack continue, exposing the company to higher risk. We must quickly find the source of the probe.

As we print the logs to retain as evidence, we trace the attack to the firm from which it's originating. The company is a major financial institution that should know better than to allow its networks to be used for such an attack.

In financial services, reputation is more important than reality - our business is grounded in trust. Companies with the best protection, which detect and stop attacks and then report them to show how secure they are, can get crucified in the press and by customers for having a security problem.

It's seven minutes into the incident, and now the real fun begins: We must get the technical situation understood and resolved by the people who run the remote network. All our business partners provide contact information, which is normally used to resolve financial problems. We're happy to use this information as a starting point. Unfortunately, it's out-of-date, and our first few calls are to an office no longer leased by the company in question.

We know this is going to take longer than we had hoped, so we send word to collect our senior management so we can brief them on the incident. We're lucky that our management team members are willing to be called early, rather than after everything is over. They accept that sometimes there will be false alarms, but even so, I hesitate to issue the request. Nobody wants to be the boy who cried wolf.

The Culprit

Suddenly, things start to look better. We've found a reception number for the company, albeit for an American office. The curious verbal dance begins: I try to convince the receptionist and various IT people that something serious is happening, requiring immediate action, without raising their fears that they are being socially engineered into helping a hacker.

(Security managers note: If you run a security team, why not provide your receptionists with simple instructions on how to deal with strange requests about security incidents that might be originating from your network?)Time is still pressing, and the log-in attempts continue thick and fast. Looking at the list of attempts, there are a few accounts on the list that aren't standard but do ring a bell. I've seen this list before. I try to remember where, as I bounce from IT team to IT team, ending up with one on the right continent.

I do my best to sound relaxed and confident as I explain for the fifth time that we're seeing unusual behavior from a machine on the partner's network: Could I get assistance in resolving the issue? Without answering me, the person at the other end leans away from the phone and shouts across the office, "Are we scanning Europe or the Far East?"

I imagine this is the feeling hackers get when they manage to get into a well-protected system. This rush of relief and adrenaline makes up for the false alarms at 3 a.m., night after night. I also realize where I've seen that list of accounts before: the brute-force list in Atlanta-based Internet Security Systems Inc.'s Internet Scanner product.

Within seconds, the attempts stop and a rather embarrassed audit team at the remote site tries to explain why its test of a DMZ (see glossary) for third-party connections included a sloppy attempt to hack our machine. A swift dual investigation uncovers the fact that they are using Network Address Translation devices that make some of our systems appear to be part of their internal network. So, despite the rather obvious "go away" banners on all our systems, they included these addresses within the range to be scanned. (I've never been convinced that these long-winded legal blurbs reduce risk, but I suppose they can't hurt.)Everyone relaxes as we receive confirmation from the remote organization that this is an authorized audit of their servers, and I brief my management. Twenty-seven minutes after my pocket shook with the first warning that an incident was occurring, I'm having my hand shaken by our manager. He's happy we've proved to a key customer that we have adequate security, and he has a great tale to poke fun at that company's CIO when they next play golf.

If only every incident had such a happy ending.