Review legacy code: Waking dragons is risk worth taking, says Trainline ops head
Checks found three potentially outage-causing scripts
Continuous Lifecycle For a business that processes 120 customer transactions per minute, the fear of IT outages is never far from the mind of Trainline's head of IS operations, Dave Stanley.
Just 15 minutes of downtime can have a serious impact on revenue. Last year Trainline.com turned over £2bn in train ticket revenue, making it the fifth largest e-commerce site in the UK, Stanley told delegates at the Continuous Lifecycle London event yesterday.
However, the organisation has 14 million lines of code which make software issues inevitable, he said. Half of its 300 staff are techies. "We also interface with a lot of background systems that are archaic," he said.
"There are dragons out there – if you are working on a legacy code base, or even existing code base."
This became apparent when his development team deployed a new feature which triggered two software uninstalls and customer booking volumes plummeting to zero. Twice.
"Off the back of this I did a trawl across our code base and found three similar scripts that would have caused an outage," said Stanley. "We understood this stuff is going to happen, so we take a pragmatic approach. Bad stuff happens, but don’t panic."
He said the company has learnt a lot in the last few years since embarking on its "DevOps journey". The business now has around 50 software deployments per week.
"We fell into DevOps," he said. Stanley took the move to "decentralise" the sign-off of product developments into production from being centrally controlled by himself. As such, he deployed key members of his operations team into the dev teams.
The reality is that the business is still divided to some extent between devs and ops, but better communication is improving improving processes.
"[You need to] talk to the devs about how the service is hosted and works in production," he said. ®