A critical flaw in the way that the AOL's instant messaging client displays Web-based graphics could be exploited by criminals to create a self-copying worm attack, security researchers are warning.

The flaw was discovered by researchers at Core Security Technologies, which has been working with AOL over the past few weeks to patch the problem. AOL's servers are now filtering instant messaging traffic to intercept any attacks, but the company has yet to patch the underlying problem in its client software, security researchers said Tuesday.

The flaw has to do with the way the AOL Instant Messaging (AIM) software uses Internet Explorer's software to render HTML (Hypertext Markup Language) messages. By sending a maliciously encoded HTML message to an AIM user, an attacker could run unauthorized software on a victim's computer or force the IE browser to visit a maliciously encoded Web page, said Core Chief Technology Officer Ivan Arce.

This type of flaw could be exploited to create a self-replicating worm attack, according to both Core's Arce and Aviv Raff, a security researcher who on Tuesday reported that he'd discovered a related flaw in AIM. "The frightening thing about this vulnerability is that it can be easily exploited to create a massive IM worm, because it doesn't require any user interaction," Raff said via instant message.

No attacks based on these flaws have been reported

Computer worms were front page news in 2003 and 2004 when Blaster, Sobig, and Sasser gummed up PCs around the world. And while Web-based worms have been in the news, a widespread instant messaging worm would be something new.

Arce says that the safest thing is for AIM users to either upgrade to the 6.5 beta code or to downgrade to a 5.9 version of the software, which does not support the HTML rendering capability.

But Raff offered different advice, saying that AIM 6.5 is still vulnerable to the flaw he discovered. According to him the best thing would be for AOL to "fix the underlying code," although AIM 5.9 is "probably not vulnerable to this specific vulnerability," he said.

AOL may not be planning to fix its AIM code any time soon, according to a statement provided by the company late Tuesday. "We have resolved all of the issues presented to us by Core Security within all past, current and future versions of AIM," AOL said without commenting on Raff's findings.

Users should still be concerned, however. Core's Arce said that he was worried that hackers could find ways around AOL's filters. What AOL has done is "just one portion of the solution and that's not the most effective portion," he said. "The most effective solution is to run a client that doesn't have the problem."