Red alert! Google's terrifying Borg machines to assimilate cluster-wrangling Omega code
Googlers tell all to our pals at The Platform
Google is ready to fold its advanced Omega cluster management software into its monolithic Borg, the technology that schedules workloads across the web giant's data center empire.
The California biz has been working on developing the Omega scheduler as an heir to the decade-old Borg for a few years now. Both are designed to turn the millions of compute cores and petabytes of storage run by the advertising giant into a quivering quicksand of resources upon which Google services and internal applications run.
All this is spread out over server warehouses spanning the globe, so juggling it is not an easy task; Omega and Borg are supposed to intelligently automate the provisioning of resources behind the scenes, so opening your Gmail inbox, or a file in Google Docs, just works and is instantaneous.
John Wilkes, a distributed systems guru at Google, told our sister site The Platform that Omega will fill a few critical gaps in Google's infrastructure.
Borg is described as a monolithic scheduler that is tricky to reconfigure with new policies, effectively tying up developers unnecessarily. Omega, as a rewrite, is supposed to overcome that: it will help Google scale out its operations in a more developer-friendly and flexible manner, we're told, although Wilkes said Borg wasn't running into any particular limits.
"We have always managed to scale Borg well," he told The Platform this week.
"We haven’t hit the scalability wall yet, so that is not the reason why we continued development with Omega. It was more about giving our developers more flexibility and also more manageability. Adding flexibility was the key factor that drove us to create Omega in the first place."
Crucially, it appears Omega will not completely replace Borg, which Googlers will continue to refine: "Leaving the Borg innovations over the decade on the ground in favor of a new paradigm would not be useful," notes Platform coeditor Nicole Hemsoth.
For the full interview, and background information and papers on Omega and Borg, check out this article right here. ®