eBay backs Flash new boy for storage

Nimbus = smaller, cheaper, cooler

Application security programs and practises

Out of nowhere, Nimbus, an all-flash array supplier, has beaten incumbents NetApp and 3PAR at eBay, with 12 arrays and 100TB of flash installed to accelerate eBay's server virtualisation work.

Nimbus CEO Thomas Isakovich said: "It's one of the largest single deployments of flash anywhere... This is a seven-figure account for us... It's something I've personally been working on for six months."

Open the Pod eBay doors Hal

Before Nimbus came onto the scene, eBay had VMware ESXi v4.1 with thousands of hosts, and NetApp FAS iSCSI and 3PAR SAN storage arrays, with about 3PB of installed storage capacity in total. It wanted to roll out standard "Pods" for virtualisation and these had to serve VMs on demand to thousands of hosts.


The Pods are are standard combinations of servers, vSphere, storage and networking: converged IT stacks in effect. The eBay guideline was that the smaller and more power-efficient a Pod was, the better. It chose Dell blade servers and Cisco networking. Who would supply the Pod storage?

NetApp and 3PAR both bid for the project, with Nimbus bidding its S-class array in 10TB capacity form. NetApp bid a system using its FlashCache (PAM) controller caching technology.

HDD and flash array price equivalence

Michael Craft, eBay's manager of QA systems admin, said: "Nimbus delivered a solution based entirely on flash memory, not a hybrid approach, at a price point comparable to primary disk arrays and with no software licensing fees to boot."

According to Nimbus figures, its S-class cost less to buy and much less to run than a NetApp array configured with the same amount of usable capacity; Isakovich saying: "NetApp has about 20 per cent file system overhead, so you need 18TB of raw capacity."

Nimbus CEO: "We can justify flash right now because the acquisition cost is less."

The acquisition cost for a 10TB Nimbus S-class with a year's support was $129,536 while the 10TB usable NetApp system was $135,168, based on 450GB, 15,000rpm drives. The purchase of years 2-5 support was $90,112 from NetApp and $45,056 from Nimbus, giving a five-year fixed cost for NetApp of $227,908 and $174,920 for Nimbus, $52,988 less.

Nimbus then added in expected power usage and cooling needs, giving out these numbers:

  • NetApp: 13,140 kWh power usage, 44,834 BTU for cooling.
  • Nimbus: 2,190 kWh power usage, 7,472 BTU for cooling.

The five-year total cost of ownership was much lower with Nimbus, even without taking performance into account. Here, according to Nimbus, it creamed NetApp, producing 800,000 4K IOPS versus the FAS array's 20,000. The cost/IOPS was $0.22 for the S-class and $11.39 for the NetApp array.

Isakovich says 3PAR and NetApp bid caching and tiering-based products whereas Nimbus bid a primary storage array, and was able to do so because it has a Halo O/S feature set including deduplication; thin-provisioning; security through encryption; snapshots; replication; and high-availability through synchronous mirroring.

He said that the S-class (below) delivers consistent performance, whereas the others depend on flash cache hits for performance, and if the data isn't in cache, performance slows.

Nimbus S-class array

He added: "Our products took about half the floorspace of the equivalent NetApp product and we use less power... We can justify flash right now because the acquisition cost is less."

Also: "I'm not even considering deduplication. We're doing it pound for pound of usable capacity. The dedupe is money in the customer's pocket."

S-class refresh

The Nimbus S-class now has 8Gbit/s Fibre Channel (to 8 ports) and 40Gbit/s InfiniBand (to 8 ports) added to the existing 1 and 10GbitE host connectivity.

The system uses 24 flash modules, not standard off-the-shelf solid state drives (SSDs), with different flash densities per module to provide the 2.5, 5 and 10TB capacity points.

Isakovich said: "We have developed new flash modules which deliver about two and a half times the performance of our first generation modules."

The S-class also has a new 144Gbit/s mid-plane with a fully non-blocking architecture so that all 24 flash modules get full bandwidth and there's no over-subscription. There are also 12 processor cores instead of the previous 8.

These changes boosts overall S-class gen 2 performance about three times over the gen 1 S-class. It can deliver 80,000 4K IOPS with a 50:50 mixed read and write workload, and 8GB/sec throughput with the same workload mix.

Up to 25 S-class enclosures can be stacked in a rack to get to 250TB of capacity, inside a single file system.

The new S-class is now shipping with sales and support in North America, Europe and Japan. The pricing is straightforward: $24,995 for a 2.5TB S-class; $49,995 for a 5TB one; and, you guessed it already, $99,995 for a $10TB one.

Thin end of a great big fat wedge

In the overall eBay scheme of things, 100TB of flash compared to 3PB of spinning disk isn't much to write home about, but write home you should. This is the thin end of a mighty, mighty wedge.

Up until now flash arrays have basically competed on the idea that they blow spinning disk arrays away on performance, although they do cost a lot more on a $/GB basis. Nimbus's numbers show it can blow NetApp and 3PAR away on the IOPS and latency fronts, and compete on the cost of the usable storage capacity as well. That's a significant marketing shift, with Nimbus claiming: "15k disk is dead; flash takes over now."

Flash is changing everything.

Three-bit multi-level cell flash is coming, which will lower $/GB flash costs and make all-flash arrays more affordable.

If customers like eBay, McAfee, Lockheed Martin and the US DoD, are satisfied that a relatively unknown and small vendor's flash array beats NetApp and 3PAR on performance, endurance, five-year cost of ownership, and has satisfactory availability, data protection and support features – right in the virtualisation sweet spot of the enterprise market – then every spinning disk vendor is at risk.

When Huawei-Symantec, Nimbus, Violin, TMS, Solid Air, SolidFire, WhipTail, or others with all-flash array product, come calling on HDD array customer accounts for new virtualisation projects the incumbents can no longer, if they ever did, take business for granted. Flash is changing everything. ®

Eight steps to building an HP BladeSystem

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
SHOCK and AWS: The fall of Amazon's deflationary cloud
Just as Jeff Bezos did to books and CDs, Amazon's rivals are now doing to it
Amazon Reveals One Weird Trick: A Loss On Almost $20bn In Sales
Investors really hate it: Share price plunge as growth SLOWS in key AWS division
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
prev story


Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.