The stand-alone performance of an individual storage device may blow away the competition, but that says nothing about the reliability of the system when it's dropped into a network, right?
Well, things get a bit fuzzy there, according to the Storage Performance Council (SPC), which recently issued its SPC-1 storage performance benchmark. SPC-1 is the first multiplatform toolset that allows vendors to measure two key factors of a storage system: how many I/O operations it can sustain and how responsive it is under moderate load.
The SPC wants storage buyers to use the new benchmarks to whittle down a short list of storage product choices before making purchasing decisions. But EMC Corp. blasted the SPC more than a year ago by dropping from the organization's ranks and charging that the benchmarks didn't represent the reliability of products in real-world networks. Even SPC officials struggle to defend the absence of components, such as switches, in the benchmarking.
Nevertheless, SPC-1 has the undeniable merit of creating a single yardstick for storage solutions. SPC-1 is first a load generator that simulates the typical I/O demand of transactional applications while measuring the response of the storage system. The benchmark also provides information about the latency, measured in milliseconds, of the storage system under moderate load. Obviously, solutions with minimal latency will complete serial transactions faster, a much-needed characteristic for interactive, chained transactions typical of applications such as order entry or order fulfillment.
Interestingly, the test bed to run SPC-1 benchmarks does not include databases or application servers, but focuses on the pure I/O performance between the host computer and the storage system, which makes for easier and more consistent comparisons across different configurations.
Watch out though. IBM Corp., LSI Logic Corp., and Sun Microsystems Inc. applied the new benchmark to drastically different products, and although results were impressive, the real-world performance of the products will likely be hindered by all that darn stuff you have to connect to it.
Another thing: It's important to understand that SPC-1 is a measurement tool for block I/O performance in a transactional environment only. Its results should not be extended to different contexts such as managing multimedia files nor be used to evaluate NAS solutions. After all, performance is only one tile of the complex storage puzzle.
SPC-1 will become more useful as more vendors publish their benchmarks on the SPC Web site. Significant comparisons will then be possible. In the meantime, when a storage sales pitch calls, ask the vendor when it intends to publish its SPC-1 results. After all, having a metric to compare storage solutions is in the best interest of your company.
Does benchmarking move you? E-mail us at firstname.lastname@example.org and email@example.com.