Git push origin undo-my-last-disaster
When it can be described, it can be automated
"I'm about to make a change that will probably wipe out all of our systems."
That was the message a developer employee once delivered to Alexis Richardson, CEO of Weaveworks, just prior to an attempted system update at his company.
Richardson, who also serves as the Technical Oversight Committee Chair for the Cloud Native Computing Foundation, recounted the tale at Continuous Lifecycle London on Thursday.
There was a click as the code change was deployed, followed by an "Oh" and some swearing, he said of the instance.
"It was a bit of a catastrophe," revealed Richardson. "Then the whole team sprang into action and in about 40 minutes, the whole system was back up again."
That incident, he said, was what got him excited about GitOps, adding that recovery today would take about five minutes.
Perhaps you've heard of DevOps, a concatenation of development and operations. Well, GitOps is more or less the same thing, but governed by declarative configuration data stored in Git, the widely used open source distributed version control system for software development.
Because Weaveworks' system is spelled out configuration files, it can be resurrected on demand without heroic effort.
"Basically everything we run is already described in Git," Richardson explained. He added that it's not just the cluster state that's stored, but the machines, how the monitoring works, the dashboards, and so on.
GitOps, at least as described by Richardson, depends on Kubernetes. The container orchestration technology ensures the desired state of an IT system – as detailed in version controlled configuration files – gets recreated with containerised machines and as much state as can be captured in those terms.
GitOps has also been called high velocity CICD for Kubernetes, where CICD stands for continuous integration, continuous delivery/deployment.
Kubernetes, Richardson suggested, has changed the game for IT teams. He pointed to a Weaverworks client Qordoba as an example. The San Francisco, Calif, marketing biz, which uses machine learning to optimize website localisation for brands, adopted GitOps practices for its Kubernetes-based microservices. In so doing, it managed to shave about 60 per cent off its production bug fix time and about 43 per cent off its response time for customer requests.
"If you're willing to cede control to Kubernetes, it can orchestrate the running of your app fo you," said Richardson.
GitOps in short is automation for cloud native infrastructure. Richardson's message is that everything that can be described can be automated, even though, as he observed, some operations should remain under manual control.
At present, not everything can be described in configuration files, a shortcoming Richardson said exists in the application layer.
He acknowledged that Kubernetes isn't easy and this level of automation may not be within reach for those who haven't jumped on the K8S bandwagon. Nonetheless, he recommended it.
"It's a bit of a beast," he said. "But it does what says on the tin." ®