Secrets to successful beta-testing

In 15 years at Case Western Reserve University in the U.S., systems engineer Jim Nauer has been involved in six major beta tests. In each case, Nauer was motivated by the failure of existing technology to meet Case's technology or business needs. For example, in the mid-1990s, the university beta-tested a native version of Novell NetWare that ran on Apple's PowerPC platform. Nauer was excited about the product's potential.

"It meant guaranteed hardware compatibility and plug-and-play device drivers, and in those days, there was quite a bit of voodoo involved in getting Ethernet and SCSI cards configured correctly," he says. "You didn't just pull [the network operating system] out of the box and click through a wizard to install it."

The beta product worked beautifully, he says. Unfortunately, Apple decided not to develop it commercially. "It would have been really neat if they'd finished debugging it and it solved some of our issues," he says. "But it didn't come to pass."

As Nauer has found -- during the Apple beta and his other more successful tests -- there are numerous benefits to being a beta tester. The most important is the ability to get your hands on cutting-edge technology to gain a competitive advantage.

However, be prepared to deal with the frustrations, the resource demands and the possible disappointments that accompany the experience. Not only does beta testing stretch the resources of IT staff members, but it also can try their patience, as they are, after all, working with buggy products. Vendor approaches to beta-testing can range from very organized to haphazard, and your own testing and bug-reporting process has to be rigorous.

As Nauer found with the Apple beta, there's no guarantee the product will ever hit the market. Still, Nauer says Case Western came out a winner. "It enabled us to get our hands on NetWare 4.1 code before we had the PC native version," he says. "We got knowledge and experience we were able to apply elsewhere."

In fact, other beta testers report that even when things go wrong, they still feel the pros outweigh the cons. Reasons include early access to new technology that can solve long-standing problems, the opportunity to influence product development and a direct line to code engineers.

"You're doing it because you have a vision; you want to beat the guy down the street and improve your revenues quarter over quarter,'' says Louis Barton, executive vice president at Cullen/Frost Bankers in San Antonio.

Secrets to successful beta-testing

The secrets? Don't expect beta-testing to be a free ride, and be prepared to jump into it with both feet, says Mark Moroses, senior director of technical services at Maimonides Medical Center in New York.

Moroses has been involved with several beta tests, and while he says his team is very accomplished at the process, it's not something he'd turn into a hobby. "We only go down the [beta test] road if we believe the technology is superior," he says. "If it's a tie, we prefer to work with an established platform with less bugs and support issues."

Moroses counts just one time that a beta test came close to being a failure. He was asked to test a thin-client version of a product he was already using in-house. Unfortunately, the beta version wasn't ready for prime time.

"As our debugging process went on, it became apparent that their approach needed to be completely redone," he says. His team pulled the plug on the beta effort and submitted feedback to the vendor. But Moroses still contends the experience wasn't a waste of time because it gave the team deeper insight into the technology's capabilities.

Join the newsletter!

Error: Please check your email address.

More about AppleBillionCiscoCognosCognosEdge TechnologyHISLeftHand NetworksNovellPLUSVerizon

Show Comments