Business

Policy

FYI: That Hawaii missile alert was no UI blunder. Someone really thought the islands were toast

False text probe reveals screw up after screw up

By Kieren McCarthy in San Francisco

110 SHARE

The individual who sent an emergency text to everyone in Hawaii warning them of an imminent missile attack did not hit the wrong button as first claimed – and was actually convinced a real attack was happening.

That's according to a report published Tuesday by America's comms watchdog, the Federal Communications Commission (FCC). Written by the regulator's cybersecurity advisor James Wiley, the dossier notes that the individual in question refused to talk to Wiley, but that he or she did write down their recollection of events shortly after they occurred on January 13, and Wiley was given a copy of that statement.

Previous to the report, the assumption was that the alert had been sent in error, and focus turned on the Hawaii Emergency Management Agency's terrible user interface on its computer systems.

It was claimed an official clicked on the wrong item in a drop-down menu. Rather than perform a test of the software without warning citizens, the agency worker accidentally selected the option to emit a real missile alert.

Hawaiian fake nukes alert caused by fat-fingered fumble of garbage GUI

READ MORE

Now it turns out there was no accidental user-interface blunder. Now we're told confusion arose when conflicting messages were sent in a test of the system during a shift change. The person at the controls thought Hawaii really was going to be wiped off the map.

"At 8.05am, the midnight shift supervisor initiated the drill by placing a call to the day shift warning officers, pretending to be US Pacific Command," the report notes. "The supervisor played a recorded message over the phone. The recording began by saying 'exercise, exercise, exercise,' language that is consistent with the beginning of the script for the drill.

"After that, however, the recording did not follow the Hawaii Emergency Management Agency’s standard operating procedures for this drill. Instead, the recording included language scripted for use in an Emergency Alert System message for an actual live ballistic missile alert. It thus included the sentence 'this is not a drill.' The recording ended by saying again, 'exercise, exercise, exercise.' Three on-duty warning officers in the agency’s watch center received this message, simulating a call from US Pacific Command on speakerphone."

As the report digs deeper, screw-up after screw-up is revealed.

One more thing...

For one, there was no system in place for dealing with a false alarm. Which seems pretty shortsighted considering the enormous importance of a ballistic warning system.

There was also a critical miscommunication between supervisors when they took over from one another at 8am on that fateful day. The leaving supervisor told the incoming day-shift supervisor that he intended to carry out a preparedness drill, but the incoming supervisor assumed he meant for those ending their shift, not the new people starting their shift that he was overseeing.

As a result, the day shift supervisor "was not in the proper location to supervise the day shift warning officers when the ballistic missile defense drill was initiated" – which is probably code for he was sat on the toilet.

Also noteworthy is the fact that the organization's policy and related checklist for the alert system had only be finalized one week earlier, on January 5.

Not only was the system new but managers decided to push it to its limits – simulating a live ballistic missile defense drill, with no notice, specifically as the shift changed at 8am. It was a worst-case scenario test – and it failed, resulting in over a million people believing that they would shortly be hit by a nuke.

It took 38 minutes for another alert to be sent telling Hawaiians it was a false alarm.

The critical error, according to the person who hit send – the day shift warning officer – was that he or she heard the phrase "this is not a drill," but did not hear "exercise, exercise, exercise." As such, the staffer thought it was a real event.

On their computer, they selected the template for a live alert – which offers a drop-down menu that includes the option for both a live alert and a test alert; a design that people have been quick to point out is less than optimal. The official chose live test and then when prompted with the message "Are you sure that you want to send this alert?" – which is also the exact same message and prompt that appears during a test – clicked yes. And out the alert went.

Is that right?

However, it is also possible that this version of events is also untrue, and the warning officer simply screwed up first by choosing the wrong option, and then refused to pause when given the warning prompt. He or she could simply be protecting their job and reputation.

The report goes with the official version – of a misunderstanding – although it inserts a few skeptical notes. "Because we've not been able to interview the day shift warning officer who transmitted the false alert, we're not in a position to fully evaluate the credibility of their assertion that they believed there was an actual missile threat," it notes, adding: "But it is worth noting that they accurately recalled after the event that the announcement did say 'This is not a drill.'"

As for the long delay in announcing it was a false alarm, that is another series of cockups. The warning officers realized almost instantly that they had wrongly sent a real message telling Hawaiians they were about to be bombed.

The drill was started at 8.05am with the call pretending to be from US Pacific Command. The alert was sent just two minutes later, at 8.07am. And then, just sixty seconds later, the mobile phone of the warning officer went off – "distinct audible tones that announce a wireless emergency alert."

That was when the rest of the team realized a live alert has actually gone out beyond their internal network. The rest of the team said they knew it was a drill so it's safe to say it was a brown-pants-moment for pretty much everyone.

The first thing they did – within the next 60 seconds – was call the governor of Hawaii to tell him it was a false alert. As we now know, he tried to send out a tweet telling people not to panic but he didn't know his Twitter password.

Then, at 8.10am, they called Pacific Command and the Honolulu police to tell them there was no missile launch. At 8.12am a cancellation is run through the system but that isn't able to recall messages or warn people that the original message was false, and by then everything is already in meltdown – the Emergency Management Agency (EMA) starts calling TV and radio stations to get the message out but its phone lines become clogged as the public try to find out what is going on.

Bunch of twits

A few minutes later, at 8.20am, some bright spark suggests using social media and the EMA posts on Facebook and Twitter that there is "NO missile threat to Hawaii." The governor retweets the message and soon after posts his own message on Facebook. But the vast majority of Hawaiians are still unaware that it is a false alarm and are in panic mode.

It's not until 8.27am – 20 minutes after the false alarm was issued – that the EMA decides it has to put out an alert using the same system it used to issue the warning –text messages to everyone's phones.

A supervisor logs into the system, but there is no template for a false alarm correction so he has to create one, get everyone's agreement that it is crystal clear, and then hit send. It finally goes out at 8.45am.

Overall "a combination of human error and inadequate safeguards contributed to this false alert," the report concludes.

It has a few observations: "Most importantly, there were no procedures in place to prevent a single person from mistakenly sending a missile alert to the State of Hawaii," it notes. Amazingly, there is – or was - no double-check system in place before a missile alert was sent to more than a million people.

The decision to run a no-notice live drill on a shift change is also met with raised eyebrows. "While other emergency management agencies use no-notice drills under special circumstances, their common practice is to schedule drills in advance for a set date and time," the report notes. In other words, whose stupid idea was that?

The report slams the fact that the software "did not differentiate between the testing environment and the live alert production environment" – the height of poor UI design. An operator is also able to send an alert by simply click "yes" on a single warning box. There are no additional login requirements.

Stupid

What's more, that exact same warning box, with the exact same language, appears when running a test. So there is little or nothing to indicate to an operator the gravity of what they are about to do.

The report notes: "Common industry practice is to host the live alert production environment on a separate, user-selectable domain at the log-in screen, or through a separate application. Other alert origination software also appears to provide clear visual cues that distinguish the test environment from the live production environment, including the use of watermarks, color coding, and unique numbering."

And then, of course, the fact that there was no way to quickly backtrack only made the situation worse. "The Hawaii Emergency Management Agency had not anticipated the possibility of issuing a false alert and, as such, had failed to develop standard procedures for its response," notes the report.

Oh, and in the aftermath of the mega-gaffe, it emerged that, during a press tour of the EMA's headquarters in 2017, sticky notes with passwords written on them were attached to agency computer monitors, and were clearly seen in Associated Press photographs.

In other words, the whole thing was an omni-shambles.

As to making sure this never happens again: all supervisors must now get advance notice of any tests; two warning officers will now be required to login and approve every alert or tests; and a false warning template has been created so if it all goes wrong again, at least people will only fear for their lives for about 10 minutes rather than 40 minutes.

The EMA will update its software to include much clearer signs that a warning officer is going to send a real live alert, and the agency will not run any more tests until its own investigation has concluded.

The FCC probe is not over – this was just a preliminary report – and it has recommended that several meetings between relevant parties should be held to discuss what can be learned from the abject failure of a critical system.

Still, on the plus side, there was no actual missile, and hundreds of thousands of people were not under immediate threat of death. Or were they? ®

Sign up to our NewsletterGet IT in your inbox daily

110 Comments

More from The Register

Hawaii Live-Go! Microsoft launches Honolulu admin tool for cloud and on-prem

One tool to rule them all

Hawaii governor: I wanted to tell everyone nuke alert was fake – I just forgot my password

Dem to be granted access to Twitter profile ‘in an emergency’

Microsoft's System Center successor edges closer to Hawaii 1-0

Project Honolulu gets a second Tech Preview

What do Cali, New York, Hawaii, Maine and 18 other US states have in common? Fighting the FCC on net neutrality

Attorneys General go to court to rescue internet protections

Gang-briefed by IBM bosses in Hawaii? Nah, I'll take redundancy

Sun, sea and solutions on the Pacific island for the Best of Big Bluers

IBM's tops sales staff face long haul...to Hawaii

Missed the ax? Then don’t miss the flight

Hawaiian fake nukes alert caused by fat-fingered fumble of garbage GUI

You'll Pai for this, thunders FCC

Fines for crossing roads while TXTing enacted in Honolulu

Hawaii really is paradise now

US MARTIAN FLYING SAUCER tests above Hawaii: postponed

Come on NASA, pull your bloody finger out

NASA scrubs FLYING SAUCER over Hawaii ballocket mission

Wrong kind of wind grounds Mars lander airbag test