Hey British coders: DevOps – you're doing it wrong

Plus you need new metrics. Chin up, let's get this sorted

If you want a brief summary of DevOps, it goes like this: a lot of those who claim to be implementing DevOps aren't getting it right. And British companies are doing worse than their peers abroad.

That’s the potted findings of a CA Technologies earlier this year that claimed there exists a gap in perception, pointing out that 84 per cent of UK organisations agreed it is important to have IT and business alignment in relation for DevOps, but just 36 per cent already had this goal in place.

But this distinction is jumping the gun. One of the main elements of DevOps is the way it breaks down barriers and dispenses with some of the existing metrics used to measure IT delivery.

Given this, then, how is it possible to measure elements such as “business and IT alignment” – areas where business units may have a different idea of success from the IT teams.

It’s a bit like asking people how much they’re drinking: they fill in questionnaires saying they’re moderate imbibers when it reality, they’re knocking it back like George Best on a Saturday night. People aren’t always good at self-assessment and can’t be relied upon to deliver accurate figures.

Georg von Sperling, senior director a CA Technologies, reckons we need to develop a whole new set of metrics when it comes to DevOps deployment.

“Traditional metrics are useless,” Von Sperling told The Reg.

The usual indicators of what marks success are outdated while, in other cases, there are elements that companies shouldn’t be measuring at all, according to Von Sperling. Examples include "vanity" metrics such as lines of code produced or function points created as these reward the wrong type of behaviour. This has seen organisations pit development teams against each other, a practice that will not lead to better code while other determine success by measuring the meantime between failures. According to a Puppet Labs’ survey on the state of the DevOps “high-performing IT teams” achieve far better stability than lower-performing peers, with 60 per cent fewer failed deployments and a mean time to recover (MTTR) that’s 168 times faster.”

However, this appears to be a by-product of the way that DevOps improves the production process. The MTTR should not be a metric in itself but as more DevOps gets rolled out and the teams become better performers.

Think of it as a football team buying better players – there’s a higher percentage of accurate passes made and more shots on target: it doesn’t mean anything in its own right but does eventually lead to more goals.

Alanna Brown, Puppet Labs’ marketing manager, agrees: “Our hypothesis is that change/fail rate doesn’t matter.” You can run quick feedback loops and automated testing as DevOps practices become more widespread.

The question is: what does work when it comes to assessing the success of DevOps projects. CA’s Von Sperling reckons you need to adopt more radical methods – but then he might, given CA is promoting its software as tools for use with DevOps. That aside, though, he does reckon you need look beyond traditional software metrics: this could, for example, mean looking at staff happiness. It doesn’t mean seeing if they’re grinning all the time but rather looking into transfer requests and staff turnover. In addition, you should have the figures to hand so that it’s actually reviewable.

It’s important that metrics aren’t corruptible - in other words, ensure the data isn’t linked to something like a financial incentive. It’s no good looking into the number of test cases produced in a month if there’s a monetary reward driving it.

These are all very different metrics to take into account and organisations have to prepare themselves for change.

An enterprise architect who has been responsible for rolling out Puppet within an organisation is Jonathan Fletcher of Hiscox. He agrees that developing a set of meaningful metrics is a difficult proposition, particularly when it comes to some of these more social interactions.

“How can you measure the empathy improvement between developers and operations staff?” he asks, acknowledging the challenge. “However, you can start to measure some of the efficiency gains such as time to release a change into production or the number of regressions introduced in a change.”

Fletcher says Hiscox is a long way from perfection but reckoned it’s made a “good start” in some areas. “For example, in one application we’ve reduced the cost of a release by 98 percent. Another example is a new product we launched a couple of weeks ago. The release took one and a half hours compared with the three or four people working around the clock that it used to take.”

There’s another element to this: the reaction of third parties. It’s one thing to shift perception within your own organisation but there’s a need to react to business partners, too.

“I’d say the most difficult part of changing culture has been influencing third party vendors and external contractors. They have their own company culture and it might not match yours,” Fletcher told us.

“My personal definition of DevOps is a culture of empathy, shared goals and incentives: with incumbent third parties it’s very difficult to align their goals and incentives to yours. If you have a third party developer come in then largely his/her expectation is to write code, not to worry about the build system or to help with deployments.“

Clearly, implementing a DevOps project has a clear path to determine its success. To work effectively there needs to be a major adjustment within your organization’s culture and you need to adopt a brand new set of metrics. But therein lies the rub. It’s become a cliché to talk about “business alignment with IT” but when it comes to DevOps deployment, it’s vital to get those metrics right.

Deciding and setting those metrics are an important consideration - not just for the business types in your organization, but you and your techie peers, too.

There is a problem – something Hiscox’s Fletcher acknowledges: hype around the term DevOps. “I have some sympathy for those who say DevOps is smoke and mirrors,” he says. “But it’s not something mystical. The challenges that DevOps hopes to address have been around for ages. It’s just got a name now and some more mature tooling and processes around it.

The geeks, it seems, will need to develop more of a business head and to start thinking about new metrics and new ways of measuring the things they deliver and the way they deliver it. ®

Biting the hand that feeds IT © 1998–2018