Did object storage come of age in 2013?
Some crossed the chasm. Some turned back...
Object storage entered 2013 pretty much as a niche technology but exited the year headed towards mainstream status, with ViPR, Black Pearl and EVault leading the way.
Others, though, held back. As of the year end, we saw IBM doing nothing much with objects, NetApp apparently standing still with its StorageGRID, and Dell exiting the field.
Counterbalancing these three, EMC re-entered the object world with ViPR, while SpectraLogic, Seagate and EVault all entered object storage in significant and different ways. Suppliers gained funding and strengthened products and, suggesting growing respectability, IDC introduced an object storage marketscape with a supplier rating eye-candy chart.
Being objective over EMC
EMC was a major mover in the object space, stirring its pot with ViPR, the provision of a separate control plane for storage hardware and software resources that store data and provide storage resources. ViPR first slithered into view in January as EMC’s Bourne Project - starring Amitabh Srivastava, head of EMC’s advanced software division and ex-Microsoft Azure Cloud man. EMC was an object storage pioneer with Centera and its Atmos system is its current mainstream object storage hardware array.
ViPR info emerged through the year with this object-tinted snake emerging from EMC’s basket of goodies during the summer.
ViPR enables data services to be layered onto its control plane with the plane translating those data services requests into commands understood by the data-storing and delivery resource of choice. Srivastava’s team included an object data service alongside more basic file and block access data services.
ViPR and its object data service
It makes little sense for a ViPR object data service to be fulfilled by an Atmos array; you save time and cycles by having apps needing object services talk to the array direct. Attention became focussed on Isilon scale-out filers being used for object storage, with a ViPR Object data services entity as a front end. Rumours emerged of energetic discussions and perhaps even some dissension inside EMC about this, but they seem to have gone way.
We have not come across any customer using the ViPR object data service yet but EMC’s energetic re-entry into the object space with these advanced concepts strengthened the general perception that object technology was becoming a major strategic consideration for EMC.
Dell, Amplidata and Caringo
How odd it was, in comparison, that Dell discontinued its OEM deal with Caringo to provide object storage software for its three year old DX6000 storage appliance line. This deed was done in April and indicates, with hindsight, that Dell couldn’t satisfactorily see into object storage accounts, not that those accounts didn’t exist. Judging by the activity of new entrants and object startups in the year they very much did exist. Dell just couldn’t find a way of reaching them profitably.
Better news for Caringo came in June when social discovery site Massive Media replaced its Amplidata object software with competitor Caringo’s software because it could handle the large number of small files better.
Amplidata in turn was fortified by Quantum’s strong focus on its OEM’d Amplidata software product, branded Lattus, which has become a major part of its StorNext file management and virtualisation suite for oil and gas, media and entertainment and other industries.
We heard in April that the Bezos behemoth, the grand-daddy of cloud object storage, had two trillion objects in its S3 storage cloud, double the number it had a year before. It stored just 10 billion objects in October, 2007. We’re probably looking at 4 trillion now, if the recent rate of progress is extrapolated roughly.
S3 stores data as objects in buckets in a key:value scheme. The details of Amazon’s object storage scheme have not been revealed. With Amplidata, Caringo, Cleversafe, EMC, HDS and Scality all growing it's apparent that object technology is not just the preserve of hyper-scale web sites like S3, and ordinary enterprises can use the technology.
Cleversafe and IDC
Cleversafe revealed a $55m D-round of funding in August. It was also named as top object storage dog in an IDC marketscape study.
IDC object storage marketscape.
This showed the top five object storage vendors, in terms of their capabilities and strategies to be:
- Data Direct Networks
In market share terms (the size of their circle on the IDC chart above) the ordering of these five suppliers was from EMC in first position, through Cleversafe, DataDirect Networks, Amplidata and then Scality in the fifth position.
This chart was illuminating not just in its contents but in that IDC reckoned the object storage market was mature enough to be worth tracking.
In December Cleversafe updated its products with faster processors, solid state storage, and memory caching and refreshed software, capping a good year.
Coho Data unwrapped
This start-up emerged from its stealth invisibility cloak in November with a $25m B-round for the continuing development of a “lash-tuned scale-out storage architecture designed for the private cloud that delivers unparalleled performance at public cloud capacity pricing.” With its scale-out DataStream arrays customers will be able “to build their own high performance Amazon-style storage for their data.”
These implement an object store with, we understand, file (NFS, SMB) and iSCSI block access protocols layered on top. HTTP-based key/value APIs will also be supported.
One interesting aspect is that the Datastream technology does not use erasure coding. The Coho Data use of object storage is individual but, all in all, it is part of the trend to stick object technology underneath file access protocols, with Coho having iSCSI block access layered on top as well.
DataDirect Networks? WOS that then?
Object storage is said to be good for data sets that will blow a file system away, so DDN announced a WOS 7000 array that can hold trillions of objects. The basic building block is a rack with 2.5PB capacity, and up to 256 of these can be clustered together. DDN said, “A cluster of 32 WOS cluster nodes can be architected into a single namespace consisting of over 32 trillion unique object IDs without slowing performance."
But that’s small potatoes in DDN-land with a 1EB federation of WOS 7000 clusters tantalisingly close, with DDN saying such a federation would store up to 983 PB and 32 trillion objects in a single WOS namespace. There’s scale and then there is SCALE.
Very few organisations today need that amount of capacity but it must be comforting to know the headroom scaling up to 32 trillion objects is there.
DDN also laid off 58 people in 2013 when a 2012 web-scale customer scaled back purchases after giving DDN a bonanza 2012.
HP's 8800 state
HP made object storage progress late in the year with its Ibrix-based StoreAll system.
In October HP CEO Meg Whitman said that the StoreAll object storage system was one of HP’s top storage priorities, after StorServe (3PAR) and StoreOnce (deduplicating backup array).
Two months later HP delivered on this with a new StoreAll system, the 8800, which replaced the previous 9320 and 9730 systems. It was powered by HP’s latest eighth generation ProLiant servers and scaled out to 16PB with 1,024 nodes. (Perhaps DataDirect would class this as a starter system.) It was certainly more powerful than the previous systems and featured extraordinarily fast search, providing an answer to what’s changed in half a billion files in 1.4 seconds compared to a trad (NetApp) filer array taking up to 40 hours. Somebody’s holding metadata in memory or flash I should think.
This 8800 supports NFS, CIFS and SMB, NDMP, HTTP and HTTPS, WebDAV, FTP and FTPS, the StoreAll REST API File Compatibility Mode and Object Mode. Native OpenStack Swift and Keystone are also supported. This file access support became fairly typical of object storage during the year - see Scality below.
Object Storage may well be better than file storage, with no need for RAID overhead and RAID rebuild times for example, but customers do not want to give up their familiar file interfaces.
HP’s 8800 will become available this quarter and we’ll see if its mix of features propel it up the object storage sales rankings.
- In June Joyent unveiled its Manta Storage Service, object storage in the cloud, with strongly consistent writes, highly available reads, no object size limits, and per-object replication policies. Joyent CTO Jason Hoffman described is as, “It's S3 with EC2 and Elastic MapReduce bolted on top of it.”
- Intel introduced its massive data, Large Object Store (LOS), asserting that "the Massive data usage model using LOS will be the foundation for all long term storage." It developed a LOS reference architecture, working with Quanta and Amplidata, and intended for use by ODMs, OEMs and ISVs. Yeah, yeah, Intel’s marketing trumpet blows hard and strong. Since then we’ve heard zip from chipzilla about object storage.
- August saw Scale Computing announce its SCRIBE project with an object-based web-scale storage architecture that assumes failures will happen. Scale Computing says it eliminates the need for any protocols.
- In November Nasuni was granted a patent for an invention which layers a filesystem onto a public cloud storage provider’s object storage systems. Once again we hear the object storage beneath a file system song being sung again.
What about Seagate, OpenStack and EVault?
What’s a disk drive-maker doing in a article about object storage? It’s a user. Seagate’s Ethernet-addressed Kinetic disk drives are accessed by applications using object style Gets and Puts and store data as objects accessed using key-value pairs. The first Kinetic drive should become available in mid-2014.
Seagate subsidiary EVault, a cloud backup vendor, introduced its Long Term Simple Storage (LTS2) service which is also object-based. This was a major initiative by EVault and stores its data on spun-down Shingled Magnetic Recording Seagate drives. (Seagate Kinetic drives may well be used in the future)
The system has an 8EB capacity objective, which EVault said was needed to achieve its pricing objectives.
The data is stored using OpenStack Swift technology; a major win for OpenStack after so much cross-industry effort has been put into it. EVault is taking on Amazon and its tape-based Glacier archive vaulting service with an offering that costs half as much again as Amazon’s ten cents per GB per month but with much faster data retrieval. LTS2 is a massive and costly bet by Seagate/EVault that it is possible to take on Amazon as a cloud storage supplier and win.
In fact to do this, to take on Amazon, EVault is saying you need object storage technology. If it does succeed then OpenStack’s Swift gets a massive boost. If it fails then Swift adoption will be slower.
As for Scality...
Scality had a great 2013:
- SGI started OEM’ing its RING product in January.
- It linked in Hadoop to the RING technology in March plus a plug-in for Cinder, the block storage layer within the OpenStack project.
- In October NFS and CIFS access protocols were added to the ring. So, like HP’s StoreOnce 8800 with its mass of interfaces, the RING had interface proliferation too, supporting block storage for OpenStack via Cinder, REST/HTTP protocols such as Amazon’s S3 API, CDMI, NFS, CIFS, AFP and FTP as well as FUSE. All data written through the REST API or file interface can be read through any of the other interfaces, and can be analysed in place using Hadoop framework.
Scality said its October “Release 4.2 offers customers the broadest range of native information access available from any object storage provider.”
SpectraLogic’s Black Pearl
Perhaps the year’s most unexpected object storage announcement was SpectraLogic’s unveiling of Black Pearl which could be loosely described as a way of backing up objects on tape using the Gets and Puts of an S3-style interface and avoiding expensive and complex backup software.
Objects age like every other piece of data and wouldn’t it be nice to put aged, low access rate objects on to cheap tape and off expensive disk? Well, yes, it would, and SpectraLogic provided a way of doing this, joining together two large scale data storage modes that had been seen as either:or alternatives for the bulk, long-term storage of data.
Black Pearl ship from Pirates of the Caribbean film.
Spectra’s Deep Simple Storage Services, DS3, are used to stick objects in buckets, and DS3 is S3 plus four more commands. Buckets are collections of data and their metadata. The Black Pearl appliance is a 2U, solid state tape library cache which turns a tape library into an object store. Spectra claimed; "Object stores don't come close to our cost/GB." Customers with Black Pearl and a Spectra tape library "can own their own [Amazon] Glacier."
We’ve already seen Verizon adopt HDS’s Content Platform (HCP) as the object storage basis for a file sharing and syncing and cloud storage service from Verizon. That potentially vaults HDS into the front rank of object storage suppliers. It provides a great reference account because HDS is the only mainstream object storage supply selected for the role in a potentially massive cloud storage service that could eventually rank alongside Azure, Amazon and Google. They have all rolled their own object storage services.
We can expect more announcements from Scality, which is probably the most determined developer of its object technology amongst all the startups. Hopefully we should hear about customers using EMC ViPR's object data services and HP might reveal a StoreOnce customer or two too.
Let's chance our arm and say while in 2013 object storage sorted crossing the chasm, in 2014 object storage will arrive as a mainstream and accepted storage technology. ®