Microsoft's SQL Server 2000 database has suffered a double embarrassment. Record TPC-C benchmark numbers that Microsoft had touted for five months were scratched from the record books after the Transaction Processing Council (TPC) found the results "noncompliant". And IBM published numbers twice as high as Microsoft's with its own DB2 database.
At the launch of Windows 2000 in February, Microsoft chairman and chief software architect Bill Gates crowed about the numbers, which showed SQL Server 2000 and Windows 2000 processing 227,079.15 transactions per minute (TPM-C), the highest ever measured. The benchmark, which beat high-end RISC servers from IBM and Sun Microsystems, was achieved with 12 Compaq Computer ProLiant 8500 servers.
But at a meeting of the TPC on June 29, the benchmark "was found noncompliant to our policies", said TPC chairman Jerrold Buggert. "Given the nature of the problem, the tests need to be redone."
The benchmark hit by Compaq and Microsoft was challenged because the tested configuration didn't allow for the primary key of a distributed database to be updated. The TPC wouldn't identify the challenger, citing the confidentiality of the challenge procedure.An Oracle spokeswoman refused to confirm or deny that the company was the source of the challenge, saying Oracle doesn't comment on TPC proceedings. The challenge briefly put Oracle on top.
Steve Murchie, group product manager for SQL Server at Microsoft, said the TPC's rules are ambiguous on this issue. He claimed that earlier benchmarks by Oracle and Compaq's Tandem Computers unit have also failed to implement primary key updateability.
According to Murchie, Microsoft has modified the SQL Server 2000 code to implement the required feature, so it will be included when the product ships.
Microsoft will rerun the tests with the updated SQL Server 2000 code and publish a new benchmark in about a month, Murchie said.
It isn't unprecedented for benchmark results to be nullified in this manner, said Jonathan Eunice, an analyst at Illuminata. "You are policed by your competitors," Eunice said.
The SQL Server-on-Windows 2000 benchmark would have lost its first-place ranking last week anyway, with IBM's publication of a record 440,879 TPM-C result for DB2 running on its NetFinity servers.
The IBM benchmark does return Win 2000 to the top of the TPC ranking, since it was run using Windows 2000 Advanced Server.
Eunice said TPM-C benchmarks for clusters aren't the most important numbers when compared to symmetrical multiprocessing (SMP). "From a manageability perspective, people don't run a 16-way cluster if they can get a single SMP server [with the same performance]," he said.