Getting the F... acebook out of Kea: Zuck's open-sourcerers retrain load balancer as a server
Replaces industry standard with homegrown DHCPLB code
Antisocial media giant Facebook has published the source code for the latest version of DHCPLB, adding server functionality to the tool that was first developed by FB engineers for hardware provisioning and load balancing.
The updated open-source DHCPLB can be downloaded from GitHub.
At Facebook, the tool has already replaced an older implementation of the provisioning stack based on Kea, an open-source DHCP server project managed by the nonprofit Internet Systems Consortium (ISC).
"With this version, we've seen better throughput and are able to iterate faster than we could with our previous solution," Pablo Mazzini, a production engineer at Facebook, wrote in a blog post today. "In fact, we are now handling the same volume of traffic with 10 times fewer servers."
Facebook has been using the Dynamic Host Configuration Protocol (DHCP) to provision its servers since 2013. The protocol, as most Reg readers know, is typically used to allocate IP addresses to machines within a network.
In 2016, Facebook released DHCPLB, its open-source implementation of a load balancer for DHCP. The proof of concept was developed during a hackathon by a team of three, including an intern. It was written in Go, a programming language created at Google, supported A/B testing, and offered a choice of two balancing algorithms.
DHCPLB and Kea were then combined into a single system that has been used by Facebook for the past three years.
The latest update to DHCPLB adds the ability to assign IPs directly, bypassing Kea. Mazzini said the main problem with the old setup was the fact that Kea is a single-threaded server – it can only process one transaction at a time, and at Facebook's scale, this resulted in "frequent bottlenecks". The company's infrastructure footprint was growing so large it was forced to constantly add more Kea servers.
Since DHCPLB was written in multi-threaded Go code, it is able to serve multiple requests simultaneously.
"The open source DHCPLB project is the framework we use for our own implementation. It now provides an interface for writing your own handler to craft DHCP responses and make calls to a back-end system," Mazzini explained.
Facebook is a surprisingly active member of the open-source community: the US tech giant says it released 153 new open-source projects in 2018, and a total of 474 since being founded.
Some of its most notable contributions include FBOSS, the Facebook Open Switching System that powers the company's top-of-rack switches, the Open/R network platform, and the PyTorch machine learning library.
The social network is also responsible for one of the most interesting experiments in the application of the open-source philosophy to hardware – the Open Compute Project (OCP), which was started by Facebook in 2011. Research by IHS Markit states that at least $2.56bn of OCP-branded kit was sold in 2018 – and the number excludes any money made by the members of the OCP Foundation's board, such as Facebook, Intel, Microsoft and Rackspace. ®