The High-Performance Computing (HPC) User Forum is establishing supercomputer testing standards that it says will serve as a valuable tool for scientific, engineering and government users.
Addressing the SC2000 supercomputing conference in Dallas this week, officials from the HPC User Forum reported progress on a plan to create better performance tests for the most powerful class of computers. The forum represents leading supercomputer users.
Forum officials said improved tests are needed to advance scientific research, industrial engineering and classified government work, all of which rely heavily on supercomputers. The forum is organized by International Data Corp. (IDC) in Framingham, Mass.
Tests need to measure not only the speed of the processing and the amount of data processed, but also the agility that supercomputers demonstrate in processing data from variable sources, said Joyce Tompsett Becknell, director of the compute platforms and architectures group at Aberdeen Group Inc. in Boston.
Testing With Care
"Testing, in general, only demonstrates that you're very good at manipulating a test," Becknell said. "You have to be very careful what you get out of benchmarks."
For example, Becknell said, in this week's presidential election, the wait for results may have made Wall Street antsy, which in turn would have put a tremendous burden on the high-end systems processing transactions for financial institutions. Those requests come in from various points over the Internet at different times; traditionally, supercomputers have handled batch data from set points at set times, she said.
"If you look at what's going on with high-end computers, we're increasingly trying to use them in the Web universe," Becknell said. "It doesn't mean that [existing tests are] deceptive; they're better than nothing. At the same time, what users use computers for is not what they're tested for. Look for tests that do loads [similar to what] you're going to do."
Although existing testing mechanisms "are nearly meaningless beyond chest-thumping publicity, in the absence of anything better, they continue to be emphasized in many procurements," said Debra Goldfarb, IDC group vice president for worldwide systems and servers.
Robert Lucas, head of HPC research at the U.S. Department of Energy's National Energy Research Computing Center at the Lawrence Berkeley National Laboratory in Berkeley, Calif., is leading the team that's developing a new benchmark suite.