High performance object storage: Not just about reskinning Amazon's S3
Read and it weep, you young'uns
Comment There are three tiers of storage: Primary storage, or block; secondary storage, or file; and object storage comes third. Object storage is immensely scalable, cheaper, durable... and slooooow. Could that be changing soon?
Let me talk about on-premises object storage here. Many end users start with an application (sync and share, for example) and then add more and more and more ...
Most of them don’t need the level of scale usually found in service providers, and many object storage system now support tiering mechanisms to slower repositories or cloud services.
Adding flash memory to these small (5/10 node) clusters becomes fundamental to delivering an acceptable performance.
More developers use object stores
Young developers started their careers using Amazon AWS services and the likes. VMs have ephemeral storage and, for them, persistent storage is S3. If you ask them to develop a new application, they ask for an S3 service.
Application repatriation and hybrid cloud is another trend. With large organizations moving some applications back to their data centres, we’ll be seeing more and more pressure on object storage systems. Some applications need more performance than others – but, again, that's not always the scale.
More use cases for object stores
Scale-out NFS/FS is becoming more important as a front-end for object storage, but not only. The types of workloads that are accessing object storage are growing too. It’s interesting, for example, that PureStorage FlashBlade supports NFS and S3, while support for other protocols will be added in the future.
Big Data applications are starting to take advantage of S3 too … why not have it faster? Again, some object storage vendors have tiering – and the same is true also for workloads such as content distribution; most accessed objects can be served from flash.
Speeding up objects and closing the circle
Up to now we’ve always had slow object storage systems and different forms of cache deployed on specific gateways or directly through the application. Flash, if any, was deployed only in small quantities to manage metadata.
This approach has the best $/GB but, depending on the application, complexity and data consistency could be an issue.
With modern object storage platforms, a first tier of flash is not difficult to implement – and, for some use cases, even an all-flash object storage system could be a good idea. A few weeks ago I had the chance to talk with an end user who has deployed an all-flash Ceph cluster.
Some primary storage vendors already have an object-based architecture at the back-end. For them it is just about mapping S3 protocol to the front-end. Yes, they are realising this is a need – and it won’t take long. ®