Netflix lets free simian software for cloud chaos
Angry ape kills virtual machines at random
Streaming video provider Netflix has released Chaos Monkey, its homegrown tool that's designed to boost the resilience of cloud-based applications in the bluntest way possible: by knocking them down.
"Do you think your applications can handle a troop of mischievous monkeys loose in your infrastructure?" asks Netflix's Cory Bennett and Ariel Tseitlin in a blog post. "Now you can find out."
The way Chaos Monkey works is conceptually fairly simple. It runs as a service on Amazon Web Services (AWS), where it seeks out Auto Scaling Groups (ASGs) of virtual machine instances. When it finds one, it picks one of its virtual machines at random and terminates it.
At first blush, this may sound like the most maddening piece of software ever, and if a hacker figured out a way to use it maliciously, it could probably cause someone some real headaches.
But Chaos Monkey is a tool, and the reason it runs around your network like a psychopathic ape is because in reality, system failures are one of the most common types of problems the people who manage cloud services must deal with in everyday life.
The point isn't to pull the plug on virtual machines for the fun of it. The point is to ensure that even though the plug has been pulled on a server or two here and there, the overall system is resilient enough to keep running anyway.
"Failures happen and they inevitably happen when least desired or expected," the Netflix developers write. "If your application can't tolerate an instance failure would you rather find out by being paged at 3am or when you're in the office and have had your morning coffee?"
Netflix has made the source code for Chaos Monkey available on GitHub under the Apache open source license. The company says it's just the first of a family of tools it calls the "Simian Army" that it plans to release to the public.
Like Chaos Monkey, the others – including Latency Monkey, Conformity Monkey, Doctor Monkey, Janitor Monkey, Security Monkey, 10-18 Monkey, and the unnervingly-named Chaos Gorilla – are all designed to root out unseen problems in cloud architectures.
The company says Janitor Monkey, which searches for unused resources and disposes of them, is the next likely candidate for release.
But even these tools can't guarantee 100 per cent uptime for cloud-based appplications. During the large-scale AWS outage in June, Netflix was knocked down along with several other customers. Still, Netflix reps say they're confident that the company's rigorous resiliency testing, using the Simian Army among other tools, is the right approach.
"We take our availability very seriously and strive to provide an uninterrupted service to all our members," Netflix developer Greg Orzell wrote in a postmortem of the outage. "We're still bullish on the cloud and continue to work hard to insulate our members from service disruptions in our infrastructure." ®
It doesn't bypass security, it just randomly shuts down virtual machines you already have complete control over. The worst that could happen is that a programming error allows the user of the tool to shut down too many servers.
Well I suppose the very worse thing would be a security flaw which would allow a 3rd party to hijack the tool while its in use.
Really, this is a smart move. Any programmer, system designer, etc. knows that their code must be tested. But when it comes to distributed systems, all to often the "plan" now is to write up some failover code and hope it works. Even the likes of Amazon themselves clearly get this wrong (since they've already had a time or two where a localized failure caused wider-scale cascading failure.) This can allow for a failure at a known time, and while someone is looking closely at the logs to make sure not only that it works, but that it's working the way it's intended to, isn't driving load up dangerously on remaining systems (before a spare instance can be spun up) and so on.
I clicked on this story thinking it was about a new build of Ubuntu...
Looks like a really useful set of tools, but...
a) how long before they get used maliciously?
b) being free to aquire, it's a lot cheaper than the current encumbent product, Fuckwit Monkey, which has a dependency on the non-FOSS "Salary" feature, HOWEVER, if used inconjunction with said encumbent, it is potentially much more expensive from a TCO point of view than either product used in isolation