Virtual lab limitations

Still, virtual test lab management tools won't be enough to stop the environments from going awry, some say. IT organizations need to define what can be tested, approach it with best practices and ensure anything tested on the VMs also is run in tests on physical machines before hitting production.

With that in mind, Sister's of Mercy Healthcare uses a virtual environment for one of three stages of tests and always completes a test run on physical machines before going live.

"We deploy an application package to production workstations in a pilot stage prior to going live. This way we make sure the software meets all the requirements and doesn't hit any snags specific to a physical machine and performs as expected," Neubauer says.

Edward Christensen, director of technical operations at Cars.com in Chicago, say he refrains from load or performance testing in the virtual test environment. "We limit our virtualization to functional and integration testing only. And unless your production environment is also virtualized, it shouldn't be used in performance testing," he says.

Others agree that performance testing, such as that for application load and availability, is not suited for virtual test labs.

"You can't ramp up the number of concurrent users to see how well the application will perform under of a load of say 10,000 concurrent users. Virtual machines do share some resources with the physical machine, no matter how few, that would cause those types of performance tests to be in accurate," Forrester's Schwaber says.

Gary Chen, senior analyst at Yankee Group, says he encourages clients to adopt virtualization for test environments because "if they do, their lives will get much easier and they will get more testing for less money." But he also warns IT professionals not to fall victim to the promise of virtualization without heeding some simple advice.

"No one should rely fully on a virtual environment for testing. Physical testing is still a must," Chen says.