Oh my God, 911 is down. Quick, call… aaargh!

AT&T's network last night was a fiasco – don't worry though, the FCC is on the case

AT&T suffered an America-wide outage of 911 emergency calls Wednesday evening, sparking some degree of panic and a swift response from US comms regulator, the FCC.

The full scale of the problem is not yet known, and AT&T has yet to put out a response beyond a tweet acknowledging the problem and another one a little under an hour later saying the emergency service had been restored.

Initial reports that the failure was limited to a small part of the country were quickly undermined by reports on Twitter of people from all over the US registering their surprise and anger.

"I just switched to @Verizon. My wife almost died!" tweeted one unhappy customer, William Spazenelli‏.

The news sparked rapid responses, highlighting the dangers of the service going down, even for minutes. But details were few and far between.

"NENA is aware of widespread reports of some 9‑1‑1 callers receiving fast busy signals," said the National Emergency Number Association, which is the industry organization that deals with 911 policy and technology, in a statement. "We are working closely with government and industry partners to understand the scope and causes of this outage."

Investigation

FCC chair Ajit Pai put out a statement Thursday morning that the federal regulator would launch an investigation into the outage. "Every call to 911 must go through," said Chairman Pai, who also noted that he has spoken to AT&T CEO Randall Stephenson about the problem.

AT&T has said in response to the FCC announcement that it "takes our 911 obligations to our customers very seriously and will be sharing additional information with the FCC."

Although 911 outages are rare, they are not unheard of. In April 2014, 11 million Americans in seven states lost 911 service for six hours following a coding error.

The FCC opened an investigation and five months later reported [PDF] that a third-party emergency call routing service named Intrado had arbitrarily capped the number of emergency calls it would accept. Over time, the number built up and when it hit 40 million calls, the company's software stopped assigning calls. In the end, it affected 61 call centers in Washington, North Carolina, South Carolina, Pennsylvania, California, Minnesota and Florida.

The FCC reached a $16m settlement with CenturyLink, a $1.4m settlement with Intrado Communications, and Verizon paid a fine of $3.4m for the outage.

Just a few months later, in August 2014, T‑Mobile suffered two 911 outages that together lasted three hours. Again the FCC investigated and fined the mobile company $17.5m for failing to provide timely notification of the outages to 911 call centers. The FCC found it had not implemented "appropriate safeguards in its 911 network architecture."

As such, it looks as though AT&T may also be on the hook for a multi-million-dollar fine, especially if it failed to communicate sufficiently quickly about the problems.

Next gen

NENA has used the outage as an opportunity to push next-generation 911 services run over the internet (IP) rather than mobile networks.

"This significant and unfortunate outage highlights the immediate need to transition America's 9‑1‑1 centers to robust and resilient 'Next Generation 9‑1‑1' technology," the organization said in a statement. "NG9‑1‑1 can intelligently route around outages, redirect calls to other regions, or use backup facilities in ways that legacy E9‑1‑1 systems cannot."

It's not the first time NENA has made the same point. In the aftermath of the April 2014 incident, NENA CEO Brian Fontes pushed for more IP‑based systems: "The next-generation system is designed for resiliency. It is designed for redundancy. It is designed to move. So, if a center goes down, the traffic could easily be moved to a different state or a different part of the country ... That's why I think there is a critical need to ensure federal support for upgrading our 911 systems into a next-generation 911 environment, which should address those types of issues raised by the commission."

Back then, FCC Commissioner Jessica Rosenworcel was a little skeptical of the idea, noting: "While a more centralized, IP‑based system can mean tremendous functionality, it also can mean new vulnerabilities," adding: "The kind of software glitch we had here is just an early demonstration case. Before a worse one results – or a malicious one, as part of a cyberattack – we need to get our house in order."

Just last month, however, draft legislation [PDF] was put forward to upgrade emergency systems – part of which will be to expand IP‑based 911 calls.

As for the guy whose wife's life was put in danger, things turned out fine. Seeing an opportunity, Verizon's social media responded to him asking if everything was alright. "Yes she is doing well. Glad to be with you guys now and I appreciate your support!"

Should we tell him about the $3.4m Verizon fine for the same sort of thing in 2015? ®


Biting the hand that feeds IT © 1998–2017