How Facebook flipped the data centre hardware market
The first rule of cloud fight club is...
10 years of Facebook The rise of Facebook helped to ruin the data centre hardware market, caused IBM to sell its x86 business to Lenovo, spurred HP to embark on an ambitious "Moonshot" server, and encouraged Dell to go private.
With the recent acquisition of WhatsApp, it's also adding momentum to the drive of OTT mobile messaging, which is increasingly snatching SMS revenues right under mobile networks' noses.
But most importantly of all, it propelled a scrappy group of Taiwanese and Chinese equipment makers into Western markets, and shook up the IT supply chain in the process.
Facebook, marking its 10th anniversary this month, has had this effect because of the way it designs, buys and tests its data centre hardware. Unlike smaller IT companies, Facebook has for many years bought gear directly from Asian-based manufacturers.
But where rival consumer internet giants like Amazon and Google have stayed quiet about their proprietary hardware and software, Facebook has decided to be open, and this openness has shaken up the way IT is bought, sold and designed.
Open up, we're coming
This kicked off in 2011 when Zuck & Co launch a scheme named the Open Compute Project, which sought to make it easier for other large-scale IT providers to take advantage of some of the economies of scale available to Facebook, and to give Facebook a more robust IT supply chain.
This scheme was launched in April 2011, but it was not clear then whether the approach was useful only for Facebook, or for other providers as well. Several years later the scheme has attracted the rest of the industry as well, with numerous other major infrastructure operators either adopting the equipment (Riot Games, Rackspace), or publicly revealing that they have been working on similar designs themselves (Microsoft).
All of Facebook's server and storage hardware is being shifted over to open compute designs, we understand, and it is beginning to swap out Cisco switches for OCP variants as well. To date, the approach has saved Facebook over a billion dollars in equipment and maintenance costs, the company said recently.
It has also taken root in the commercial IT industry and has spawned a diverse set of businesses that are trying to make money from the designs, ranging from traditional IT giants like HP and Dell to Taiwanese component makers like Wiwynn and Quanta, to small IT services and consultancy firms like Hyve Solutions, HPC specialists Penguin Computing, and Boston Servers.
"Two years ago we only knew that it was going to be popular but we didn't really know how popular that will be," explains Quanta's general manager Mike Yang. "Since especially second half of last year I started receiving a lot of inquiries about OCP rack. The time is coming. People [have] gradually recognised that OCP provides them an easier way to choose the hardware they want."
Though Quanta does not break out the revenue it makes from OCP-style designs, it is understood that it supplies them to a variety of major cloud providers such as Facebook and Microsoft. (That said, Quanta is hedging its bets and is involved in an OCP-like scheme named "Scorpio" which draws together Intel and Chinese companies like Baidu and Tencent to make data centre gear for the Asian market.) One out of every seven servers shipped in 2012 was made by Quanta, if you believe the company's figures.
Yang reckons that OCP is going to be a strategic area for Quanta's business.
Other companies agree: US-based Hyve Solutions, for instance, has started to sell OCP-inspired designs to large data centre operators and has found its business "growing rapidly", says the company's general manager Steve Ichinaga.
"It's clear that it has definitely taken off," he says, though notes that "it is still better suited for people running at a larger scale, I don't have lots of small customers today."
Hyve Solutions, it should be said, has built its most successful OCP gear to a 19-inch rather than 21-inch rack form factor to take account of legacy data centres.
Penguin Computing, which builds Open Compute-based designs for the high-performance computing market, has gone for the full 21-inch approach.
"Facebook solved the specific problem for themselves which was to get a lot of infrastructure as fast as possible. They were constrained by the physical size of the rack. We're able to get three servers along rather than two - that in essence gives you a third more capacity in the same space," explains David Ingersoll, Penguin Computing's veep of sales.
Like Hyve, Ingersoll believes Facebook's new style of hardware is in the early stage of broader uptake, noting that, "This will be the year of adoption to the small projects, and then moving into 2015 and that [there will be an] accelerated slope curve."
Customers of the new hardware seem keen on it as well, with some caveats.
"A lot of constraints were removed by open compute," explains Rackspace's veep of supply chain operations Wesley Jess. "I think there's a whole new host of apps and environments where this could work really well as it grows up."
Adapt or die
Rackspace is in the process of adopting OCP-style designs for all of its data centres. By using the design, it can take advantage of some of the economies of scale available to much larger companies like Facebook and Google, Jess explains, while also paying less on a per-server basis than before.
However, they had to make several tweaks to the stock open compute specifications to make them fit the workloads Rackspace operated, explains Rackspace's principal engineer Aaron Sullivan: "Those systems generally didn't have the management feature-sets we were looking for," he says.
"There were lots of serviceability challenges for Knox up-front [so] we collaborated with Quanta on one design that sought to get the same kind of density but a little bit more bandwidth [and] make it serviceable so people that work in traditional enterprise DCs can be familiar with it."
But once Rackspace had dealt with some of the problems, it found the new equipment allowed it to pay less than before while upgrading hardware at a quicker rate.
It also was able to benefit the wider community, as its revamped OCP storage design made its way back into the design's progenitor, Facebook.
"We were single-sourced on OpenVault and put an RFP out for OpenVault-compatible storage," says Facebook's supply chief and OCP bigwig Frank Frankovsky. "The second source [was] already available because Rackspace had co-developed it with Quanta!"
What happens next?
Even cloud companies not involved in the scheme, like IBM's recently acquired SoftLayer cloud, recognise the influence of the scheme: "I think it will have a significant influence on our hardware going forward," explains SoftLayer chief Lance Crosby, though he said the company was likely to focus on its own non-OCP designs for the time being.
So, what does this have to do with the fortunes of IBM, HP, and Dell? By opening up the large-scale data centre hardware market to smaller companies, Facebook is effectively doing public relations for Asian equipment-makers like Quanta and Wiwynn. This has helped give them further credibility, and has indirectly likely driven their server sales to higher levels than before. Server sales revenue from these Asian companies rose 45 per cent last year, according to IDC, with more expected.
The rise of these companies has led to intense competition with established server vendors like HP, Dell, and IBM, and has steadily eroded the profit margins these companies can extract from servers and storage. This, in turn, has caused a series of cascading problems for them that led HP to shake up its server lineup by adding in new OCP-inspired servers like the "Moonshot" system and get rid of long-time exec Dave Donatelli. It also helped push Dell to go private so that Michael Dell could re-engineer the company for the brutal cloud infrastructure world, and IBM to jettison its server business by selling it to Lenovo before the margins dragged down its all-important earnings-per-share figure further.
The OCP project looks set to go further, though, given its recent launch of a networking scheme. Watch out Cisco – Facebook's coming for you. ®
Sponsored: Network DDoS protection