Feeds

Are SPEC file benchmarks broken?

Flash versus disk

Protecting against web application threats using SSL

Comment Recent EMC SPEC filer benchmarks have annihilated competing systems. EMC has used virtually all-flash systems to dramatically up its benchmark game, leaving disk-based rivals in the dust. Are the SPEC filer benchmarks now so unrealistic as to be worthless?

If you think the question extreme just look at this pair of SPECsfs CIFS results.

An EMC VG8/VNX5700 system with 112TB of flash (560 x 200GB drives) and just 6.3TB of hard disk capacity recorded an eye-watering 661,951 ops/sec. The nearest disk-based result was an EMC VG8/VMA one of 142,979 ops/sec; about four and a half times slower. The nearest non-EMC result was a NetApp FAS3210 with 64,292 ops/sec: more than 10 times slower than the flashy EMC system.

The pattern is similar in NFS land. An EMC flash-heavy VNX system scored 497,632 ops/sec on the SPECsfs2008 NFS benchmark. Next best was an IBM SONAS system with 403,326 ops/sec from 1,975 disk drives. An EMC VG8/Symmetrix run with 512 disk drives scored 135,521 ops/sec. A NetApp FAS6240 delivered 190,675 ops/sec; EMC's flashy VNX was two-and-a-half times faster.

Thoughts? Several. First, these all-flash results trounce disk-drive based ones. From now on there is no conceivable way a disk drive-based system could beat a flash-based system in the SPEC CIFS and NFS benchmarks.

Secondly, these flash-based systems are unrealistic: Ferraris compared to diesel Golfs. IBM's Rich Swain, a NAS storage specialist, says: "Benchmarks are for vendors to tweak a configuration beyond the average customer purchase. They are like a golf course with one golfer using a graphite shaft driver versus a golfer using a steel shaft club. They both drive the ball 300 yards or so, but the graphite driver goes 20 yards more. Is that enough for the extra dollars?"

In a way EMC has gamed the SPEC benchmark system, but the rules let it do so. Technology has moved on and a benchmark conceived in an all-disk server era can now be criticised for allowing unequal comparisons.

The cost of the flash in EMC's CIFS benchmark could be $3.36m assuming a $6,000 cost per flash drive. That's hardly a realistic system, but that's not the point. EMC is now top NFS and CIFS SPEC performance dog and has the bragging rights to go with that. It hopefully gets a halo effect that glamourises its more realistic VG8/VNX configurations.

Other vendors can make a strong case that the SPECsfs2008 benchmark is now no longer for for purpose, and should be split into all-flash one and all-disk versions. Alternatively, the benchmarks could have a cost element added so that the flash cost/op could be compared to the disk cost/op.

We could see vendors in general abandoning SPEC filer benchmarks because they are becoming so unrealistic, and the cost of configuring and running systems to seize the performance crown is getting so high as to be unaffordable. The exaggerated performance crown specs defeat the very purpose of the benchmark SPEC, so to speak. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
Wanna keep your data for 1,000 YEARS? No? Hard luck, HDS wants you to anyway
Combine Blu-ray and M-DISC and you get this monster
Google+ GOING, GOING ... ? Newbie Gmailers no longer forced into mandatory ID slurp
Mountain View distances itself from lame 'network thingy'
US boffins demo 'twisted radio' mux
OAM takes wireless signals to 32 Gbps
Apple flops out 2FA for iCloud in bid to stop future nude selfie leaks
Millions of 4chan users howl with laughter as Cupertino slams stable door
'Kim Kardashian snaps naked selfies with a BLACKBERRY'. *Twitterati gasps*
More alleged private, nude celeb pics appear online
Students playing with impressive racks? Yes, it's cluster comp time
The most comprehensive coverage the world has ever seen. Ever
Run little spreadsheet, run! IBM's Watson is coming to gobble you up
Big Blue's big super's big appetite for big data in big clouds for big analytics
Seagate's triple-headed Cerberus could SAVE the DISK WORLD
... and possibly bring us even more HAMR time. Yay!
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.