Manchester Arena attack: National Mutual Aid Telephony system failed
Servers were over capacity, Kerslake review finds
Vodafone has come in for criticism for the "catastrophic failure" of its National Mutual Aid Telephony system (NMAT) in today's review of the deadly suicide bombing at the Manchester Arena last year.
Twenty-two people were killed and over 100 injured in the attack at an Ariana Grande concert at the northwest England indoor arena on 22 May 2017.
The Kerslake Arena review report found that the NMAT completely failed on the night, which severely hampered the setting up the Casualty Bureau (CB). The CB is the single point of contact of receiving and distributing information about people believed to be involved in an incident and is set up by police local to an incident area.
The failure meant that 0800 numbers that should have been provided for those caught up in the attack could not be issued. It took until 0300 GMT on 23 May, four-and-a-half hours after the attack, which took place just after 2230 GMT on 22 May, before a restricted local telephone number was running.
The panel found that a “catastrophic” failure occurred, stemming from an initial inability to record a Manchester Arena-specific message over a number previously allocated to an incident due to one of the servers being over capacity
The independent panel, chaired by Lord Bob Kerslake, former UK civil service head, described this failure as a cause of “significant upset and stress” to families on the night and went on to describe a number being “reduced to a frantic search” of local hospitals to get information.
The panel found that a “catastrophic” failure occurred, stemming from an initial inability to record a Manchester Arena-specific message over a number previously allocated to an incident due to one of the servers being over capacity.
The manager of the CB contacted Vodafone, only to learn that the management of the system had been sub-contracted to an outfit called Content Guru.
The report went on to say that the Content Guru representative had little knowledge of the requirements or the role played by the CB:
The Panel was informed that the Vodafone representative had little knowledge of the Casualty Bureau arrangements and referred the Casualty Bureau manager to another company called Content Guru, to whom Vodafone had sub-contracted technical guardianship of the service.
Again, the Panel was informed that the Content Guru representative had little knowledge of the police requirement or the vital role the Casualty Bureau contributed to the response operation.
According to the report, the manager was eventually told that, of the eight servers in place to spread the load of call-taking, one was at capacity with a memory-full condition and so could not accept the new message, which lead to the message being rejected over all servers.
And thus the service failed.
By 0230 on 23 May, police had enough call-takers on site to allow the CB to go live and with the NMAT still failed, a decision was taken to use a local (0161) number, which resulted in significantly less capacity than had the national system been available.
Vodafone’s CEO has offered formal and personal apologies to the those affected by the fault and assured the Home Office that migration to a new platform will provide the necessary fall-back.
The report recommends that the Home Office ensures henceforth that it has guarantees that disaster recovery arrangements are put in place to avoid the failures that plagued the NMAT on the night of the attack.
A Vodafone spokesperson told The Reg:
As a result of a failure of one of our systems, Greater Manchester Police was not able to issue an 0800 incident response number in the immediate aftermath of the Manchester Arena Attack. This was clearly unacceptable and we sincerely apologise for the distress caused to those affected by this terrible attack.
We have been running the National Mutual Aid Telephony service for the Home Office since 2009. It provides an 0800 number and call handling solution for police forces during a major incident, and has been successfully deployed on numerous occasions. However, any failure is unacceptable and we have since undertaken a major upgrade of the system in question and it is being tested on a daily basis.
The Register has contacted Content Guru for comment. ®
Sponsored: Becoming a Pragmatic Security Leader