Leopard's firewall is confusing, inconsistent, switched off by default and incompatible with some applications, a security researcher said after analyzing the new security tool.

"This firewall is a mess," Rich Mogull, a security consultant and former Gartner analyst, said after spending two days digging into the new firewall's capabilities. "It's a step back from Tiger's firewall. I was originally pretty bullish on Leopard's security, and I still am on the concepts, but the implementation makes most of its advances ineffective or unusable."

The firewall in Mac OS X 10.5, a.k.a Leopard uses a bare-bones interface -- earlier this week, Mogull called it "so simple as to be nearly useless" -- that offers users three options:

• Allow all incoming connections
• Block all incoming connections
• Set access for specific services and applications

Other settings let users switch on the stealth mode, which is supposed to cloak all ports on the Mac, preventing attackers from even "seeing" the machine when scanning the Internet for open ports, and probing for potential victims. After a Leopard upgrade, the firewall is set to the first, "Allow all..." which means, in fact, that the firewall is switched off. Users with machines that had the firewall turned on also saw their firewall turned off after Leopard was installed.

"'Block all...' does seem to block actual connections," said Mogull, "but any shared ports are detected as 'open/filtered' on a port scan." And unless users turn on stealth, some services -- Bonjour, Apple's network device locating technology, is one -- are seen as open by scans, no matter what firewall setting is selected. Only by using "Block all..." with stealth enabled are shared services actually invisible.

"In short, 'Block all...' seems to block inbound connections but ports show as open/filtered," he said. "Stealth mode works, partially, but some ports still show on a port scan no matter what. Bonjour is always accessible, unless you're in stealth mode."

Those inconsistencies pale against the firewall's ability to break some applications without warning. While testing the firewall's "Set access..." option, Mogull discovered that Leopard prevents some applications from running.

When the "Set access...." mode is turned on, Leopard digitally signs applications that the user allows access to incoming communication. But if that application is subsequently changed -- say when it's updated to a new version -- the signature no longer matches and the application won't run. While that's typical of firewalls, Leopard also blocks applications that change at runtime. Skype, the popular VoIP software and instant messenger, is one such program.

If the user has set the firewall to "Set access..." and runs Skype, the icon will bounce a time or two on the dock, but not load. Nor does Leopard tell the user that Skype has failed or why it won't launch. Only the Mac OS X Console gives a clue, with a message such as: 11/2/07 9:47:51 AM [0x0-0x35035].com.skype.skype[399] Check 1 failed. Can't run Skype

"You can fix this by reinstalling Skype," said Mogull, "and it will work until the next time it's run. Then you have to reinstall Skype again. That's a bit of a problem."

Skype has acknowledged the problem as far back as August, but while it said then it was working on a fix, no version compatible with the Leopard firewall has been issued. (The last Mac OS X Skype update was released in July.)

"I was close to recommending the firewall in app signing mode ["Set access..."], but not with this problem of breaking applications," he said. Instead, users should rely on the protection built into their routers. "If you have a wireless router between the modem and your Mac, you're fine," he said.

But even a flawed firewall isn't fatal, Mogull noted, for Mac users. "I run a firewall, but it's kind of out of habit from my days with Windows," he said. "I like that extra layer of protection." But there have been few cases where exploits have actually claimed success against Apple's systems. "There was a zero-day, but that's been patched. I don't know of any ways of getting in today." He declined to name the exploit.

"Fortunately, all of this is fixable," he said. "Apple clearly was a little rushed, but they're moving in the right direction. It's our responsibility to keep on Apple to make sure they convert these concepts into actual implementations."