Business

So you accidentally told a million people they are going to die: What next? Your essential guide...

Hawaii's EMA – you’ve never seen crisis management quite like this

By Kieren McCarthy in San Francisco

84 SHARE

Analysis It's a common enough scenario: you are in charge of a critical piece of your company's computing infrastructure, and you make a simple mistake with far-reaching implications.

It could be a mail server misdirecting internal documents into journalists' inboxes, or a poorly configured border firewall. Perhaps you accidentally changed access permissions to a shared drive. It all happened so quickly, one mouse click, and now you and your department are being inundated with people demanding to know what happened.

What should you do? In the blur of such an event, it can be all too easy to make the wrong decisions, or even take the right actions but do them in the wrong order. When time is a critical factor, pressure can make it even harder to stay level headed.

Which is why we are all lucky to have reports on precisely what happened at the Hawaii Emergency Management Agency (EMA) – which earlier this month texted more than a million people on the US islands that ballistic missiles were about to rain down on their heads.

Those reports – one from the FCC and another from the state of Hawaii [PDF] – give a detailed blueprint for exactly how to handle such a crisis.

Their actions were exemplary and worthy of detailed study. We will break down the main points here in the hope that it can help serve in the event of your own emergency.

In the case of Hawaii's EMA, they faced a rather sticky situation: they had accidentally sent a phone alert to more than one million people telling them they would shortly die in a military attack, perhaps perishing in the radiant glow of a detonating thermonuclear device.

Not a great situation to be in, but one where clear thinking is essential to prevent things from getting worse.

So what's the first thing you should do? That's right: hand the problem to someone else.

Ass protection

The key to protecting your ass is to immediately inform the most important person you can quickly access outside of your department. It then becomes their problem.

So it was with EMA – the first call went to Hawaii's governor. The call went something like this.

"Hi Governor Ige, this is Bob from the Emergency Management Agency, we have a bit of an urgent situation."

"Hi Bob, what seems to be the problem?"

"Well, we just texted everyone a message that turns out not to be true."

"Everyone?"

"Yes, roughly one million people."

"Oh wow, ok, what did the message say?"

"Let me see… it said BALLISTIC MISSILE…"

"Sorry, do you mind not shouting?"

"Yes, sorry, it said: 'Ballistic missile threat inbound to Hawaii. Seek immediate shelter. This is not a drill.'"

"Oh I see. Not a drill, huh. WAIT… WHAT?!"

"Key takeaway here is: there is no missile. Anyway, lovely talking, David. All the best."

Work down the list

With that out the way, the EMA crack crisis management team hit the phones letting everyone of decreasing importance know what had happened. Start at the top and work your way down.

In this case, it was: the police; then the people that operate emergency sirens; then those in charge of the different counties. Then the TV and radio stations. But in a typical corporate situation that might be: CEO; head of security; anyone else in the IT department; then department heads; then your internal message board or Slack.

Once you have offloaded the problem, the next most important thing is to stop anyone else from bothering you. Most valuable in this case is social media: just post a message to your Facebook page or on Twitter stating succinctly what has happened. It doesn't matter if anyone sees it or not – so long as you put it out there, you can't be blamed for not letting everyone know.

What is important, however, is that you don't accept any sort of blame. In fact, if at all possible, imply that someone else was to blame and you are the one sorting out the mess. This will give some wiggle room later as well as allow you to ignore anyone asking questions until you've locked down a story. Any questions should be met with: "I'm too busy right now sorting this out."

The EMA really excelled in this respect. It posted the same message on both Twitter and Facebook: "No missile threat to Hawaii. False Alert. We're currently investigating." Textbook.

Ok, so the immediate crisis is over and you have some time to fix what the actual problem is. Do this as fast as you can while everyone is still saying "do you know what happened?" to one another over and over again. The faster the problem is fixed, the sooner you can focus your efforts on figuring out who or what to blame next.

Man vs machine

Pinning the blame can be tricky, and will vary every time depending on what you did wrong. If at all possible, blame a machine rather than a human. But make sure it's a machine you're not in charge of.

In the case of the EMA, they hit on an ingenuous solution: their system's user interface. The options for performing a test alert and sending a real alert to hundreds of thousands of folks were on the same dropdown menu.

Not only does this explanation immediately make sense to ordinary users, who have to wrestle crap software all day long, but it also passes blame onto whoever devised the UI in the first place – which is going to be damn near impossible to pin on any one individual.

It was a brilliant piece of crisis management, especially given the pressure they must have been under to explain why one million people had been told they were going to die.

Now in most cases, that will likely be the end of it. Your department is tasked with writing a report about what went wrong with the system – because no one else understands it – and that gives you plenty of time to make sure that no one gets fired. Plus there's an opportunity there to ask for extra resources for a few pet projects.

Be generous: spread those extra funds around and before you know it, those that know what really happened will be thanking you rather than eyeing you suspiciously.

Unfortunately for EMA the widespread panic its message induced meant that not one but two independent investigations were ordered. If that happens to you, there is only one solution: gang up on one individual and get them fired.

So long as everyone agrees to stick to the same story, the situation can actually work in your favor. Fed up with Jeff's constant bitching and flatulence? Time for him to go. The team had always had concerns about him but management didn't listen. Or maybe it's payback time for Michael, that smarmy git.

Quick and easy

So long as you pile on whoever you choose, it's easy to make it a case of "one bad egg" and everyone else can get on as normal. Management will be too terrified to dig any deeper and will go for the easiest and quickest solution.

The EMA, yet again, excelled. The Hawaii state report left no doubt that "Employee 1" was to blame – and not for the first time.

"Poor performance evaluations have been documented on select SWP individuals," the report notes. "This has created morale and competency issues. Employee 1 has been a source of concern for the same SWP staff for over 10 years."

Sorry, Jeff, or Jane, or whoever you are, you should never have eaten that last piece of cake at the Christmas bash three years ago. And just for good measure:

"Employee 1's poor performance has been counseled and documented and the SWP members have stated that they are 'not comfortable with Employee 1 as a supervisor, two-man team, or as a part of the SWP in general. He is does not take initiative and has to be directed before he takes action. He is unable to comprehend the situation at hand and has confused real life events and drills on at least two separate occasions.'"

Jim – or was it Jade? – can try all they like to argue that they didn't hear "exercise, exercise, exercise," but we all did, didn’t we, fellas?

Done. Dusted. Sorted. Now, how much extra can we stick in this year's budget for more stuff – you know, to make sure this never happens again? ®

Sign up to our NewsletterGet IT in your inbox daily

84 Comments

More from The Register

Hawaii Live-Go! Microsoft launches Honolulu admin tool for cloud and on-prem

One tool to rule them all

Hawaii governor: I wanted to tell everyone nuke alert was fake – I just forgot my password

Dem to be granted access to Twitter profile ‘in an emergency’

FYI: That Hawaii missile alert was no UI blunder. Someone really thought the islands were toast

False text probe reveals screw up after screw up

Microsoft's System Center successor edges closer to Hawaii 1-0

Project Honolulu gets a second Tech Preview

What do Cali, New York, Hawaii, Maine and 18 other US states have in common? Fighting the FCC on net neutrality

Attorneys General go to court to rescue internet protections

Gang-briefed by IBM bosses in Hawaii? Nah, I'll take redundancy

Sun, sea and solutions on the Pacific island for the Best of Big Bluers

IBM's tops sales staff face long haul...to Hawaii

Missed the ax? Then don’t miss the flight

Hawaiian fake nukes alert caused by fat-fingered fumble of garbage GUI

You'll Pai for this, thunders FCC

Hunt for Planet X finds yet another planetoid, just not the right one

Astroboffins are Goblin up Solar System sightings

New theory: The space alien origins of vital bio-blueprints for dinosaurs. And cats. And humans. And everything else

Phosphine tests hint at development of DNA, RNA