It’s the computer mistake that continues to get international attention.
So far we’ve taken a look at two different screenshots that emergency officials say staff turn to when it’s time to send out a test or real emergency alert.
Always Investigating uncovered those were just two small pieces of a much bigger puzzle.
The Hawaii Emergency Management Agency recently changed vendors last fall for its emergency alert software. While emergency officials wouldn’t say who they use now, we traced past and recent messages to AlertSense, one of many FEMA-approved vendors nationwide.
Neither AlertSense nor HI-EMA are confirming the vendor partnership, citing security, but AlertSense did walk us through a standard system, one that has fail-safe after fail-safe built in.
HI-EMA first said the user who sent the false alarm from an incoming-missile message template had screen options that looked something like blue-text hotlinks. Then HI-EMA changed the handout to one that looked more like a black-and-white drop-down menu.
First officials said one click sent the doomsday message. Then it was two clicks, with the second click to confirm.
But Always Investigating has learned it was far more complex than that, and the crisis could have been averted at many screen views along that process.
AlertSense agreed to walk Always Investigating through how its system should be used by any customer, using a generic example interface.
“The process is every bit as important as the technology,” explained Randy Grohs, CTO and vice president of engineering at AlertSense.
At the start, you can write a new message or use a locally written template, like HI-EMA did Saturday. The system forces you to double-check everything.
“The next step is to create, modify, verify, and preview the content of the message itself,” Grohs explained.
Always Investigating asked, even if someone is relying on a pre-populated template, would they nevertheless have to look at the screen with all the content before hitting send?
“Absolutely. There is no way to go through the entire process of sending an alert without visiting this screen and this form,” Grohs said, “and because of the height of this screen, you have to scroll to get to the (‘send message’) button at the bottom.”
Right above that “send message” button is a map of who you’re about to reach. So after the first click to create, the scroll to review, and the click to send, another step pops up and asks “Are you sure…” with yet another click needed.
“A dialogue shows up over the form. This is designed in a way as to allow you to continue to see what’s underneath so you can still verify the content and the message,” Grohs said.
Close to the approval button, you can still see the words of the message about to go out, and Grohs said industry-standard practice is to leave the word “TEST” as the first word of any test message.
As for quickly undoing a mess if an error wasn’t caught before all that, AlertSense says it can be done from scratch, or a quick message overwrite in the template just sent.
“Would anyone trained in the system properly know how to do that that quickly?” Always Investigating asked.
“They should,” Grohs said. “FEMA has training materials all agencies are required to take. There’s not supposed to be anyone operating these systems who hasn’t been trained.”
FEMA requires certifications from agency users like HI-EMA staff before they can use any software that sends emergency messages through the IPAWS (FEMA’s Integrated Public Alert & Warning System) backbone. Software vendors like AlertSense, which provide the systems to get messages onto IPAWS, also provide training.
Always Investigating has been asking HI-EMA for days who and how many are FEMA-certified and vendor-trained, and especially — was Saturday’s operator qualified? HI-EMA has not yet told us.
Grohs says AlertSense tells all customers that “treating every alert, whether it’s a practice alert or a live alert, as though it’s the real thing helps people get into a mindset of the seriousness of the tool they are using,” and “there are many processes that we know our customers use to ensure that human error doesn’t occur, and being prepared for what to do in case it does occur, how quickly to retract something and being practiced at that is a key.”
HI-EMA, the FCC, FEMA, and lawmakers all say they’re investigating. HI-EMA says it’s already made process and template changes. Software vendors are even using the event as a wake-up call.
“Us along with other vendors that work with IPAWS as a front-end, for sure we’ll continue to work with our customers and train and use this as a great lesson for all of us,” said Dave Smith, AlertSense president and chief operating officer.