What benchmarks can tell you about your solid-state drives

Consumer tools

Not everyone is testing virtual machines or trying for 500,000 IOPS. Many people just need to test the disks that are attached to their desktops, notebooks or servers. Here, any number of consumer tools can help.

CrystalDiskMark is a perennial favourite. Its widespread use makes it easy to find other people online with similar setups so you can compare your results.

Alexej Schepeljanski's AS SSD is another popular offering, beloved of the community because it is SSD aware. It doesn't really do much for testing spinning rust.

HD Tune Pro is worth a mention, and Futuremark's PCMark 7 will definitely offer the broad community for benchmark comparison; it is the main selling point of this tool.

Anvil's Storage Utilities has recently been making the rounds on testing sites as an up and coming tool. I haven't had a chance to really put it to the test on known systems to see how accurate it is, but many popular sites swear by it.

Atto Disk Benchmark is a bit more upscale, targeted not just at the consumer market but at RAID cards and HBAs as well.

It can be thought of as AS SSD's antiparticle: instead of being all about SSD awareness, Atto looks to bridge the gap between consumer ease of use and the sort of raw benchmark agnosticism that Iometer and its ilk offer.

Workload testing

As we have said, raw benchmarks cannot tell you how a real workload will perform. The exact configuration of the operating system matters. Patches matter. Access patterns matter a lot. Workload simulation tools are thus very important, especially to enterprises.

Under the category of "too expensive for Trevor to afford" are SPEC and SPC. I would love to have these available to test all the really neat stuff that crosses my lab, but alas, they are priced for those who have vendor backing.

SPEC Solution File Server (SFS), recently updated for 2014, serves primarily to show what kind of response times you can expect when serving network file protocols such as NFS and CIFS.

SPC: the tests of concern are SPC-1, SPC-1/E, SPC-1C, SPC-1C/E, SPC-2, SPC-2/E, SPC-2C, SPC-2C/E. The tests simulate OLTP, large file processing, large database queries and video on demand.

On the more affordable front (read free), Microsoft offers two tools for SQL. Use SQLIOSim to simulate SQL activity to validate stability, and SQLIO to do performance benchmarking.

SQLIO is a very powerful tool if you know how to use it properly.

Microsoft also offers Exchange Jetstress 2013. (2010, 2007) it is the evolution of the LoadSim tool that was used with Exchange 2003.

For testing file workloads, I use a scripted instance of Quaddra's Storage Insight on a 10M file test set that I have been building up.

The script goes in and resets the "date last accessed" to random values between one and 10 years for all files. It then indexes the entire lot of them and copies anything that hasn't been accessed in two years from file server A to file server B. Then the script restarts. It is simple and effective.

LoginVSI is worth a mention here as well. While it mostly serves to test VDI, it is evolving. Storage-heavy VDI tests can be done, and I fully expect the company to be moving towards proper server testing in the future (especially if VMware obtains clue and buys it).

A hundred companies can all use the same operating system on the same storage and see completely different results

Something like LoginVSI is the ultimate workload generator because it will actually run the workload in question. I am sure that any enterprise that honestly needs a given server workload tested could simply knock on LoginVSI's door and it would be happy to help build a simulator for that environment.

Ultimately, the holy grail of benchmarking is sticking as close as possible to the exact applications that will be deployed on the storage in question and as close as possible to the exact access patterns.

This is where raw benchmarking moves over into profiling. It is where mere bragging rights become a useful tool, and it is also where all the angst in benchmarking comes from.

My workload is not your workload. A hundred companies using Microsoft SQL can all use the same version operating system on the same storage and see completely different results because the application feeding data into Microsoft SQL is different in each instance, as are the number of users, their roles and even the time of day when they use the thing.

Raw benchmarks will teach you about your storage device. But workload simulators will teach you about how your applications will use that storage.

Both bits of knowledge are required to make informed decisions on what to buy, when to upgrade and – as is increasingly the question – how much flash you need to add to make performance acceptable. ®

* Different operating systems handle metadata quite differently. Windows Server, for example, absolutely falls off a cliff at 10M files using NTFS. Trust me on this. Contig will help you some, but over 10M files you need to move to ReFS. Right up until you hit about 100M files; then you are hooped and you need to move to Linux, Solaris or start looking into object storage.