VMware plans to open its hypervisor to security vendors with a set of APIs that make it easier to protect virtual machines from threats including viruses, Trojans and keyloggers.

Without these APIs, security vendors building antivirus and firewall tools for virtual servers are removed from the hypervisor by several layers and therefore cannot see everything that happens within the virtual environment, according to Yankee Group Analyst Phil Hochmuth.

This potentially makes security products less robust than they could be, and creates annoyances for users. For example, a customer might have to install one instance of an antivirus program on each virtual server, rather than let one instance of the program protect all the virtual machines within a physical piece of hardware, Hochmuth says.

VMware intends to fix that problem with VMsafe, the set of APIs announced Wednesday.

"Instead of installing and running McAfee [antivirus software] on 20 different virtual servers, you just do it once," Hochmuth says.

Twenty security vendors are already building products to protect virtual machines using the VMsafe APIs, according to VMware.

Christopher Bolin, CTO at McAfee, says VMsafe is intended to be part of the VMware virtual-machine hypervisor to allow third-party developers to "see network memory, CPU and traffic before the traffic reaches the guest operating systems."

By giving partners more visibility into traffic at the hypervisor level, they will be more likely to catch malware and other types of intrusions before they enter a virtual system, Hochmuth says.

Previously, security software really had no advantage over malware that's infiltrated a virtualized server, says Parag Patel, vice president of alliances at VMware. The visibility into the hypervisor afforded by the VMsafe APIs gives security software a higher degree of privilege than malware.

The APIs also improve security with more thorough isolation of virtual machines, Patel says.

The 20 vendors developing new security products for use with VMware include Check Point, F5 Networks, IBM, Imperva, McAfee, EMC's RSA division, Secure Computing, Symantec and Trend Micro. Security products built using VMsafe should be out later this year, Patel says.

While VMware has dominated the server virtualization market, its products have suffered from security problems. Last September, VMware had to fix three bugs in its virtual machine software that could be exploited to gain control of any machine running in a virtual environment.

Earlier this week, a vulnerability in VMware's desktop virtualization software (VMware Workstation, Player and ACE) was revealed by Core Security Technologies. Core researchers discovered that malicious users or software running on guest systems can gain full access to the host computer, if shared folders are enabled. VMware acknowledged the security problem and promised to fix it.

The VMsafe APIs are meant to protect VMware's server virtualization products, such as ESX Server.

Bolin says although VMsafe is not yet part of ESX Server today, McAfee has developed a security product prototype that shows how it's possible to make use of VMsafe to run antivirus scans, memory scans, and a network intrusion-protection system to protect the guest operating system environment.

Bolin says McAfee and VMware jointly demonstrated how that would work on VMware's ESX software during the VMworld Conference in Cannes, France Wednesday. "We demonstrated the ability to scan offline file images of a disk, detect and clean malicious code and use behavior-based monitoring."

Bolin says he doesn't know exactly when VMware will make the API public.

Microsoft also is developing a similar third-party security API for its virtual-machine technology, Bolin says, though Microsoft doesn't appear to be as far along as VMware is, he added.

(Ellen Messmer contributed to this report)