Understanding benchmarks

In years past, companies that evaluated databases with an eye toward a purchase usually examined the TPC benchmarks as part of their cost/benefit analysis. During this period, databases were typically hosted on a legacy system, a single database server, or, at most, a two-node database server cluster because those were the limitations at the time.

The parameters uncovered during TPC benchmarking were fairly straightforward, given the simplistic hardware configurations available and database support at the time. However, the relevance of external database benchmarks has always been debated as not representing the real world. Today, the computing architecture we use to support our database infrastructures is vastly different from the database configurations of old.

Database clustering is now commonplace, and we're well beyond the two-node limits of just a few years ago. Clustering configurations can be set up in myriad ways and can be tweaked for greater initial or long-term performance.

The database clustering configuration you choose for your company could vary widely from those used in the TPC tests. You can further shape your performance by tweaking cluster configurations and database parameters to meet your needs.

Further detracting from the meaningfulness of database benchmarks is the fact that relational technology is fairly mature now. When TPC benchmarking began, relational technology was still new, and the market demand for the software and the benchmarks was intense.

Today, it is less likely that a company with an existing investment in a particular database will abandon that database for another based merely upon TPC benchmarks. The need to compete aggressively while maintaining an extremely short time to market means most companies have a slim chance of effecting a major change to their database infrastructure.

In years past, companies also invested in multiple database technologies for specific projects. These days, those same companies are more concerned with integrating their multiple databases than with jumping ship when a hot benchmark comes along.

The motivation behind this latest benchmark may well be driven by Compaq's desire to increase interest in its high-end server platforms. Moreover, Microsoft's SQL Server 2000 is on the verge of coming to market.

Of course, that's not to say that the TPC's benchmarks are entirely without value. The numbers do highlight the strength of Compaq server clusters, so if you are currently in the market to expand your database infrastructure, you would be wise to evaluate this cluster option.

One notable problem I perceived with the testbed is the use of beta software. Using a database release that is still in beta and making the results publicly available seems questionable. Database vendors can and should measure the effectiveness of their database performance as a part of the release cycle. However, the benchmarking numbers achieved during a beta period may actually play out quite differently once the software is released.

Finally, per-transaction costs shown in this latest TPC benchmark cannot be considered completely accurate. The investment needed to upgrade to Windows 2000 is high, and the cost of running SQL Server 2000 is not yet fully known. In order to cast away such doubts, perhaps another benchmark that includes the total investment picture and uses the production version of SQL Server 2000 would be advisable.

Ultimately, the best database benchmarking can be found by running the numbers against your own real-world configuration. Performance continues to be a key part of database infrastructure, but with many more options available to boost your speed, the business relevance of external benchmarks will remain questionable.