Dropbox slips 500PB into its Magic Pocket, not spread over AWS
Shifts 90% of your files from Amazon to in-house systems
Dropbox has sucked the vast majority of its data off Amazon's cloud servers and into its own custom storage centers. This shunting of customers' files has been dubbed Project Magic Pocket.
In a blog post on Monday, Akhil Gupta, Dropbox's veep of engineering, said his company has been an early adopter of Amazon S3 to store bulk data – the firm has always kept metadata and web services in-house – but was now handling 90 per cent of its own storage needs.
"As the needs of our users and customers kept growing, we decided to invest seriously in building our own in-house storage system," he said.
"We knew we'd be building one of only a handful of exabyte-scale storage systems in the world. It was clear to us from the beginning that we'd have to build everything from scratch, since there's nothing in the open source community that's proven to work reliably at our scale."
The advantages are twofold, he said. Firstly, Dropbox can optimize its storage systems to increase performance by taking out the public cloud lag. Secondly, he claimed that the company can save money through hardware and software optimization of its storage needs.
Magic Pocket started in concept in September 2013 and by August the following year the company began shifting data over to its custom cloud operation, cutting prices at the same time. By October 2015, the firm had switched over 90 per cent of its data onto its own servers.
The results, Gupta claims, have been impressive, with claimed data durability of over 99.9999999999 per cent, and availability of over 99.99 per cent. It's all encrypted at rest and Dropbox plans to roll out more infrastructure to speed things up further.
On one level, going in-house is a sign of the maturity of the company; Dropbox is now handling over 500 petabytes of data, encrypted at rest, from 500 million customers, we're told. Running a closed shop may be able to give the company some savings and better control over costs.
But it has also got to be a fairly pricey operation to run, and with the firm still dithering about going to IPO – and investors getting twitchy about it – going in-house at this stage is an interesting move that might be scrapped if the benefits aren't as good as first thought. ®