LA air traffic meltdown: System simply 'RAN OUT OF MEMORY'
Maybe a reboot will fix it. Maybe a reboot will fix it. Maybe a reboot will fix it. Ma-
A computer crash that caused the collapse of a $2.4bn air traffic control system may have been caused by a simple lack of memory, insiders close to the cock-up alleged today.
Hundreds of flights were delayed two weeks ago after the air traffic control system that manages the airspace around Los Angeles' LAX airport went titsup as a U-2 spy plane sped overhead.
Although few details about the system crash have been made public, anonymous sources claimed to Reuters that the problem arose when a controller entered the altitude of the spook flight into the En Route Automation Modernisation (ERAM) system, developed by Lockheed Martin.
The U-2's flight plan did not include an altitude for the aircraft, but it did include a route that showed the aircraft leaving and reentering the LAX area multiple times.
As U-2s normally operate at about 60,000ft, a controller seemingly made an educated guess by tapping in that height. The ERAM system then, it is claimed, started to consider all possible altitudes between “ground level and infinity”, sending the software into a death spiral.
ERAM began spitting out error messages and then entered an endless reboot loop, which is a non-optimal state for a piece of critical equipment.
Laura Brown, spokeswoman for the Federal Aviation Administration, said the system was forced to analyse a large number of routes to "de-conflict the aircraft with lower-altitude flights". This process "used a large amount of available memory and interrupted the computer's other flight-processing functions", she added.
"The system is only designed to take so much data per airplane," an anonymous source continued. "It keeps failing itself because it's exceeded the limit of what it can do."
The same sources insisted this vulnerability did not leave the ERAM system open to attacks by miscreants. Which is handy, because it is now being used to help shepherd flights all over the US.
However, there are expected to be two talks at this year's Def Con conference which will discuss ERAM, suggesting it might not be as hack-proof as the authorities claim.
"If it's now understood that there are flight plans that cause the automated system to fail, then the flight plan is an 'attack surface,'" said Dan Kaminsky, co-founder of the White Ops security firm.
"It's certainly possible that there are other forms of flight plans that could cause similar or even worse effects. This is part of the downside of automation."
However Nate Pair, president of the Los Angeles Center for the National Air Traffic Controllers Association, said the fact ERAM was back up and running within an hour was a good indicator of the system's strength.
"We were completely shut down and 46 minutes later we were back up and running," Pair said.
"That could have easily been several hours and then we would have been into flight delays for days because of the ripple effects." ®
Sponsored: Benefits from the lessons learned in HPC