Software glitch led to London Ambulance Service outage – report
Update 18 months earlier meant recycle bin drained system resources, leading to crash
A software glitch in the London Ambulance Service’s 999 call handling system, which prevented the system’s recycle bin from being emptied, led to an outage at the New Year – according to an official report.
Control room staff were forced to log all emergency calls by pen and paper and pass information to ambulance crews over the radio when the Computer Aided Dispatch system went down for nearly five hours on New Year’s day.
The London Ambulance Service is investigating whether the outage led to a patient's death. The patient had their treatment delayed as a result of the high volume of calls and the manual dispatch system that was in place during the outage.
However, the Report on the New Year’s Day 2017 Computer Aided Dispatch system outage said it would be “inappropriate” to comment on the impact this delay had on their condition and treatment ahead of a pending inquest by the coroner.
A software update made 18 months ago was the cause of the incident, and the process to manage the recycle bin was borked and using all available system resources due to high demand on New Year’s eve. During the early hours of 1 January, 2,810 emergency calls were received, compared with 2,517 in 2016.
Chief executive of the London Ambulance Service, Garrett Emmerson, said: “This review exposed some significant short-comings in our IT processes and governance, which we have taken action to address. However, it also found that the computer system that we use to respond to 999 calls is fit for purpose.
“The system fault that occurred on New Year’s Day has been fixed and cannot now happen again. However, I would like to apologise to patients who had to wait longer than they otherwise would have for medical help on that morning, and pay tribute to our staff who responded incredibly well, ensuring patients continued to receive care during our busiest time of the year.”
Since January, the service has strengthened its senior-level IT expertise, appointing a new interim Chief Information Officer, Ross Fullerton, and non exec director, Sheila Doyle, to provide better oversight and management of the call handling system and wider IT issues. ®
Updated to add: Fullerton told The Register: “The first thing I want to say and make sure that gets across, is that patients had to wait too long and we really are sorry for that.”
He added: “The investigation that took place found the fault fairly quickly, which was reassuring. I was also really assured that a number of reports found the system itself fundamentally sound... what I picked up from my report was the need to strengthen management and IT process to make sure we have the right governance in place.”
He said the service will regularly test and maintain the system, and will strengthen the manual processes.
Asked if he believed it would take less time to get the system back online if a similar incident were to occur, he said: “I am confident processes in place are suitable for the service."
Sponsored: Becoming a Pragmatic Security Leader