EMC ‘just looking’ at benchmark test

EMC Corp has moved a step closer to becoming the last high-end storage supplier to see its hardware performance benchmarked. Within two hours of industry group the Storage Performance Council releasing its test specification for scrutiny by non-members such as EMC this week, engineers involved with the company's Clariion storage array placed an order for a copy.

Yesterday however EMC maintained its previous distance from the SPC-1 benchmark, which measures IOPS and throughput performance of disk arrays and was launched this year. It said its $2,500 order was simply part of a watching brief, and that it is still doubtful that the test represents real-world performance. "It just became available, and we said we'd stay close to it and take a look at it when it became available." The fact that it ordered a copy little more than an hour after its release was only a coincidence, the company said.

EMC's continuing refusal to embrace the benchmark is however giving its rivals increasing grounds to argue that it is running shy of the measuring stick. So far IBM Corp, Hewlett Packard Co, LSI Logic Corp, Sun Microsystems Inc and start-up 3Par data have published SPC-1 benchmark figures. Sun's test was of a rebadged Lightning array originally made by Hitachi Ltd.

According to the Council two more major suppliers are about to receive final audit approval for their tests, and more companies are in the pipeline."We're taking bets on when EMC joins in. It won't be long. The precipitating factor will be the breadth of other suppliers doing it, and the breadth of products. It would be different if it was just one or two companies, but this is every major competitor, right in their space," said Walter Baker, SPC administrator and auditor. According to Baker, some suppliers are already being pressed by customers to quote SPC-1 benchmark performance.

EMC's order for a copy of the benchmark specification came from its former Data General site in Westboro, Massachusetts, where the mid-range Clarrion was developed. While the Clariion's high-end sister product the EMC Symmetrix is widely believed to have fallen behind the Lightning - and possibly the IBM Shark - in terms of performance, the Clariion has not suffered from such judgements. EMC would not comment on ComputerWire's speculation that it will test the Clariion first, and not test the Symmetrix until the next generation of that device has shipped - expected to happen around the first quarter next year.

An EMC spokesman insisted that if the company were ever to use the SPC-1 test, it would be just as likely to measure the Symmetrix as the Clariion. "The analysis of the benchmark would be performed by EMC - not by any division," he said.

The SPC counts IBM, HP, Sun, and Veritas among its members. EMC resigned its membership last year when it said that the test then under development would not measure real-world performance, because it would not properly measure the effectiveness of caching algorithms. Yesterday it said: "The test measures a single point in an extremely wide spectrum of use. If you look at the results it measures spindle speeds. If IO requests are continually going to disk, basically that's what you're measuring." The SPC has said previously however that it used techniques patented by IBM to generate test data which it insists closely resembles a real workload.

© ComputerWire