How the anti-spam industry lost its way...
- 03 November, 2005 11:01
By PETER STEWART Chairman, New Millennium Solutions
The anti-spam industry has lost its way in trying to prevent unwanted email. That is the only conclusion I can draw from the actions of most major vendors.
One definition of madness is continually trying the same thing and expecting a different result. Yet this is exactly what much of the industry is doing with the current “solutions” to prevent spam.
Today’s rising spam volumes mean that the many and expensive approaches to stopping junk email simply are not working. Doesn’t that tell us something?
Solutions that engage the spammer in an intellectual race are doomed to failure, and the process to discover this will be long and expensive. Why? Because spammers always have the initiative; all those expensive analysts can’t do a thing until the spammers make their next move and then all the analysts can do is catch up. All that software to detect and filter unwanted email comes at a cost, as does its administration. All those processes to detect spamming IP addresses – and their administration - also come at a cost.
Some anti-spam companies, such as Trend Micro and Spamhaus deploy systems that check the IP addresses of email messages against “reputation” databases of millions of suspect IP addresses.
This approach makes little sense as both legitimate and illegitimate senders finish up on these databases. It’s not really a problem for the spammers; the fact that such databases exist means that the spammers’ use of IP addresses will be transient and being on such lists is of little consequence. However, legitimate senders cannot afford to be so cavalier with their IP addresses and once on these lists it is an issue for them as their email is stopped and they have to take steps to be removed.
Every company has one or many IP addresses and the number depends on their network complexity, i.e. redundancy, number of web sites etc. IP addresses are very important; for example, a domain name will translate to an IP address. An IP address is the equivalent of your name and date of birth, a combination that will uniquely identify you. Likewise the IP address uniquely identifies the mail server. How does your IP address find its way on to such a database and how does it get off it? Also, how do you know it’s there?
Many examples of these database systems exist, and most seem to rely on a matter of trust in the use of information. Yet how do you know, for example, if your competitor or a mischief maker might simply have submitted your IP addresses as a source of spam?
Presumably, the first indication that your IP address is on such a database is when sent mail stops getting through. If the purpose of the database is to stop suspect senders, then it wouldn’t make sense to tell the sender, would it? There is an SMTP response to alert the sender that the email is not being processed since it is suspected of being spam. A common practice is to simply ignore the request to send email, although the real issue is that even if the sender is made aware, what do they do about it? As one of our clients who is a network administrator says: “Do I try to find a contact in the company that owns the server? There are lots of these occurrences and I don’t have time for that!”
So the problem with these databases is that legitimate mail gets caught in the net. It’s easy for the spammers, of course, because they simply change their IP addresses regularly. Even if they get on the “bad guy” list, they don’t care as they’ll soon be using different IP addresses. However, the good guy? He’s not going to want to change his IP addresses, or be able to do so, hence he has to work out if he’s on a black list and decide what to do about it. Email administrators have no time to inspect the response from any and every the destination server to determine if the mail has been refused due to the sender’s IP address being on a suspect spam list, or that email is simply no longer getting through.
“It is an offence to interfere with snail mail, so why should it be any different with electronic mail?”
The end result is that all the legitimate mail senders have to take steps to bypass the “bad guy lists”, so they all finish up taking the same steps as the spammers. The task of the server should not be to determine what should and shouldn’t get through, it’s task must be to deliver mail. The social pact should be between the sender and the receiver, as it is with any mail system. It is an offence to interfere with snail mail, so why should it be any different with electronic mail?
Perhaps the only time you might want to take action against an IP address is if your email server detects unusual levels of incoming volume. The server might wish to guard against harvesting or storming, by inhibiting the traffic but in no way taking action that prevents the passage of any email – regardless of the source or the purpose of that email. I’d be furious if I thought the Post Office was stopping my mail because it felt the source to be undesirable.
Because most anti-spam products rely on the filtering approach, the challenge-response method of blocking spam – as deployed by New Millennium Solutions’ TotalBlock - is not widely understood. Challenge-response builds a list of acceptable incoming email senders by replying automatically to all those who are not on the user’s allowed list. The reply message contains an action that, when followed, automatically adds the sender to the allowed list. Since the authorisation process requires human intervention, it bypasses drone machines that spew out high volumes of spam. All address book entries are authorised automatically, as are senders who reply to the challenge sent by the user.
In challenge-response, the cost implication for the sender is for the first message only, where the sender must solve the puzzle. After that, all mail from the sender will get through. The cost implication for the receiver is that the sender may choose to not solve the puzzle. The cost implication for the spammer is that no messages will get through – isn’t that exactly what everyone but the spammers wants to achieve?
It is necessary to inspect the content of wanted messages, since they may contain an unwelcome virus. However, it’s hard to justify the effort that is put into inspecting and quarantining billions of messages that aren’t wanted in the first place.
If the industry can ever get its act together and verify the sender, then the problem will be easily managed. In the meantime, the industry should stop wasting people’s money on the spam race.
It has been claimed that the effectiveness of challenge-response diminishes when people receive 200 (new) emails a day and “asking them to respond to 200 mails a day just to authenticate is totally unacceptable.”
However, it is the sender – not the receiver - that undertakes the authentication, so it really doesn’t matter how many new emails arrive from unknown senders because it is the senders who must respond to the “human puzzle”.
It is possible that some non-spammers will send 200 emails a day to new receivers. If all those receivers had challenge-response, then the sender would have to reply to 200 challenges (“puzzles”). But who would send out that many emails to new receivers each day? Newsletters? A sales or promotion company? Such companies would most likely not object to the effort involved in solving 200 “puzzles”. As legitimate senders of email they would most likely welcome the opportunity to see their email not classified as spam and quarantined.
The intent of challenge-response is to make the sender undertake an expense, i.e. the effort to respond to the challenge in order to send email, the result of which is the avoidance of the email shot of tens of thousands of emails to new receivers. It’s hard to have sympathy for the sender incurring additional cost when the intent of challenge-response is to protect the receiver. I’m a receiver and when I got to the point of “I can’t take anymore”, challenge-response solved the problem for me. I like it because it works, because it solves the problem and because it opts out of the spam race.
# # #
For more information Peter Stewart, New Millennium Solutions (02) 9437 9800 firstname.lastname@example.org
David Frost, PR Deadlines (03) 4341 5021