eBay backs Flash new boy for storage

Nimbus = smaller, cheaper, cooler

7 Elements of Radically Simple OS Migration

Out of nowhere, Nimbus, an all-flash array supplier, has beaten incumbents NetApp and 3PAR at eBay, with 12 arrays and 100TB of flash installed to accelerate eBay's server virtualisation work.

Nimbus CEO Thomas Isakovich said: "It's one of the largest single deployments of flash anywhere... This is a seven-figure account for us... It's something I've personally been working on for six months."

Open the Pod eBay doors Hal

Before Nimbus came onto the scene, eBay had VMware ESXi v4.1 with thousands of hosts, and NetApp FAS iSCSI and 3PAR SAN storage arrays, with about 3PB of installed storage capacity in total. It wanted to roll out standard "Pods" for virtualisation and these had to serve VMs on demand to thousands of hosts.


The Pods are are standard combinations of servers, vSphere, storage and networking: converged IT stacks in effect. The eBay guideline was that the smaller and more power-efficient a Pod was, the better. It chose Dell blade servers and Cisco networking. Who would supply the Pod storage?

NetApp and 3PAR both bid for the project, with Nimbus bidding its S-class array in 10TB capacity form. NetApp bid a system using its FlashCache (PAM) controller caching technology.

HDD and flash array price equivalence

Michael Craft, eBay's manager of QA systems admin, said: "Nimbus delivered a solution based entirely on flash memory, not a hybrid approach, at a price point comparable to primary disk arrays and with no software licensing fees to boot."

According to Nimbus figures, its S-class cost less to buy and much less to run than a NetApp array configured with the same amount of usable capacity; Isakovich saying: "NetApp has about 20 per cent file system overhead, so you need 18TB of raw capacity."

Nimbus CEO: "We can justify flash right now because the acquisition cost is less."

The acquisition cost for a 10TB Nimbus S-class with a year's support was $129,536 while the 10TB usable NetApp system was $135,168, based on 450GB, 15,000rpm drives. The purchase of years 2-5 support was $90,112 from NetApp and $45,056 from Nimbus, giving a five-year fixed cost for NetApp of $227,908 and $174,920 for Nimbus, $52,988 less.

Nimbus then added in expected power usage and cooling needs, giving out these numbers:

  • NetApp: 13,140 kWh power usage, 44,834 BTU for cooling.
  • Nimbus: 2,190 kWh power usage, 7,472 BTU for cooling.

The five-year total cost of ownership was much lower with Nimbus, even without taking performance into account. Here, according to Nimbus, it creamed NetApp, producing 800,000 4K IOPS versus the FAS array's 20,000. The cost/IOPS was $0.22 for the S-class and $11.39 for the NetApp array.

Isakovich says 3PAR and NetApp bid caching and tiering-based products whereas Nimbus bid a primary storage array, and was able to do so because it has a Halo O/S feature set including deduplication; thin-provisioning; security through encryption; snapshots; replication; and high-availability through synchronous mirroring.

He said that the S-class (below) delivers consistent performance, whereas the others depend on flash cache hits for performance, and if the data isn't in cache, performance slows.

Nimbus S-class array

He added: "Our products took about half the floorspace of the equivalent NetApp product and we use less power... We can justify flash right now because the acquisition cost is less."

Also: "I'm not even considering deduplication. We're doing it pound for pound of usable capacity. The dedupe is money in the customer's pocket."

S-class refresh

The Nimbus S-class now has 8Gbit/s Fibre Channel (to 8 ports) and 40Gbit/s InfiniBand (to 8 ports) added to the existing 1 and 10GbitE host connectivity.

The system uses 24 flash modules, not standard off-the-shelf solid state drives (SSDs), with different flash densities per module to provide the 2.5, 5 and 10TB capacity points.

Isakovich said: "We have developed new flash modules which deliver about two and a half times the performance of our first generation modules."

The S-class also has a new 144Gbit/s mid-plane with a fully non-blocking architecture so that all 24 flash modules get full bandwidth and there's no over-subscription. There are also 12 processor cores instead of the previous 8.

These changes boosts overall S-class gen 2 performance about three times over the gen 1 S-class. It can deliver 80,000 4K IOPS with a 50:50 mixed read and write workload, and 8GB/sec throughput with the same workload mix.

Up to 25 S-class enclosures can be stacked in a rack to get to 250TB of capacity, inside a single file system.

The new S-class is now shipping with sales and support in North America, Europe and Japan. The pricing is straightforward: $24,995 for a 2.5TB S-class; $49,995 for a 5TB one; and, you guessed it already, $99,995 for a $10TB one.

Thin end of a great big fat wedge

In the overall eBay scheme of things, 100TB of flash compared to 3PB of spinning disk isn't much to write home about, but write home you should. This is the thin end of a mighty, mighty wedge.

Up until now flash arrays have basically competed on the idea that they blow spinning disk arrays away on performance, although they do cost a lot more on a $/GB basis. Nimbus's numbers show it can blow NetApp and 3PAR away on the IOPS and latency fronts, and compete on the cost of the usable storage capacity as well. That's a significant marketing shift, with Nimbus claiming: "15k disk is dead; flash takes over now."

Flash is changing everything.

Three-bit multi-level cell flash is coming, which will lower $/GB flash costs and make all-flash arrays more affordable.

If customers like eBay, McAfee, Lockheed Martin and the US DoD, are satisfied that a relatively unknown and small vendor's flash array beats NetApp and 3PAR on performance, endurance, five-year cost of ownership, and has satisfactory availability, data protection and support features – right in the virtualisation sweet spot of the enterprise market – then every spinning disk vendor is at risk.

When Huawei-Symantec, Nimbus, Violin, TMS, Solid Air, SolidFire, WhipTail, or others with all-flash array product, come calling on HDD array customer accounts for new virtualisation projects the incumbents can no longer, if they ever did, take business for granted. Flash is changing everything. ®

Best practices for enterprise data

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
VMware builds product executables on 50 Mac Minis
And goes to the Genius Bar for support
Multipath TCP speeds up the internet so much that security breaks
Black Hat research says proposed protocol will bork network probes, flummox firewalls
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Microsoft's Euro cloud darkens: US FEDS can dig into foreign servers
They're not emails, they're business records, says court
Microsoft says 'weird things' can happen during Windows Server 2003 migrations
Fix coming for bug that makes Kerberos croak when you run two domain controllers
Cisco says network virtualisation won't pay off everywhere
Another sign of strain in the Borg/VMware relationship?
prev story


7 Elements of Radically Simple OS Migration
Avoid the typical headaches of OS migration during your next project by learning about 7 elements of radically simple OS migration.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
A new approach to endpoint data protection
What is the best way to ensure comprehensive visibility, management, and control of information on both company-owned and employee-owned devices?