Original URL: http://www.theregister.co.uk/2013/10/08/nsa_data_centre_electrical_meltdown/

NSA data centre launch delayed as power surges 'melt metal, zap racks'

'Flash of lightning inside a 2-foot box' causes year-long delays

By John Leyden

Posted in Data Centre, 8th October 2013 17:09 GMT

+Comment Electrical surges at the National Security Agency's massive data centre in Utah have delayed the opening of the facility for a year as well as destroying hundreds of thousands of dollars in kit, the Wall Street Journal reports.

Ten "meltdowns" in the past 13 months have repeatedly delayed the Herculean effort to get the spy agency's colossal snooping facility up and running, according to project documents reviewed by the WSJ.

One anonymous project official told the paper that electrical failures created fiery explosions, melted metal and caused circuits to fail. He described these so-called arc fault failures as "a flash of lightning inside a two-foot box".

The first electrical arc fault struck the Utah plant in August 2012, according to project documents. The nine further failures since then, the most recently of which struck on 25 September, caused as much as $100,000 in damage, according to a project official quizzed by the WSJ.

Investigators took six months to work out the causes of two of the failures. As further problems occurred, the civilian contractors at the sharp end of the project hired more than 30 independent experts to run 160 tests that chewed up 50,000 man-hours – without reaching a definitive conclusion about the cause of the problem, or how to prevent it.

The causes of the meltdowns remain under investigation, but despite this the signals intelligence agency is reportedly pressing ahead with plans to fire up some of its servers this week. According to project documents cited by the WSJ, the first of the four units at the Utah centre was originally scheduled to open in October 2012.

NSA spokeswoman Vanee Vines acknowledged problems but told the WSJ that "the failures that occurred during testing have been mitigated. A project of this magnitude requires stringent management, oversight, and testing before the government accepts any building."

According to the newspaper, architectural firm KlingStubbins designed the electrical system, acting as a subcontractor to the three main building contractors, Balfour Beatty Construction, DPR Construction and Big-D Construction Corp. The joint venture told the WSJ: "Problems were discovered with certain parts of the unique and highly complex electrical system. The causes of those problems have been determined and a permanent fix is being implemented."

The US Army's Corps of Engineers is overseeing the facility's construction. "The cause of the electrical issues was identified by the team, and is currently being corrected by the contractor," Norbert Suter, chief of construction operations, told the WSJ. The army would ensure the facility is "completely reliable" before handing it over to the NSA, he added.

As recently as last week, other army engineers criticised plans from civilian contractors to sort out the electrical supply problems that have bedevilled the project.

"We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents," a report last week by special investigators from the Army Corps of Engineers warned.

Efforts to "fast-track" the Utah project bypassed regular quality controls in design and construction and meant that the weren't any clear plans about the design of the electrical system, the report said. This, in turn, ensured that the consequences of any changes to the electrical system were unclear – and made problems more or less inevitable as a result.

The installation of devices to isolate components in the event of an failure are nothing better than a stop gap, the army engineers argue. The causes of the incidents "are not yet sufficiently understood to ensure that [the NSA] can expect to avoid these incidents in the future," a report by the Army Corps of Engineer's Tiger Team warned.

The Utah facility, one of the Pentagon's biggest ever construction projects, will cost $1.4bn (excluding the Cray supercomputers it will host) and span more than one million square feet. It's become a powerful symbol of the NSA's controversial dragnet surveillance programme in the wake of ongoing leaks from NSA contractor turned whistleblower Edward Snowden.

The capacity of the project is classified but reckoned to be bigger than Google's largest data centre. Key to the successful operation of the facility will be the ability to keep electrical juice flowing to the computers, networking kit and storage equipment hosted in the facility while making sure the technology is kept cool.

Project officials told the WSJ that once up and running, the facility will draw an estimated 65 megawatts, enough to power a small town of 20,000, at a cost of more than $1m a month.

Comment

So what's going on with these apparent short circuits? If the electrical supply problems were affecting a controversial industrial facility outside the US, El Reg's security desk might be inclined to suggest a new version of the infamous Stuxnet worm – which targeted industrial control systems – is to blame.

However, reading between the lines of the WSJ story, it seems a combination of disagreements between government officials and civilian contractors over how to roll out the electrical control systems and overly tight programme schedules have have caused all sorts of problems.

The 120-acre Utah data centre is close to the intersection of two major power corridors. In Phase One of the building project, allocated a budget of $52m, the initial electrical work will be performed, including the connection of the two corridors. Later phases will include $340m in electrical work.

Cheap electricity was one of the main reasons for selecting Bluffdale, Utah, as a location for the site of what will become the NSA's main data centre. The NSA also plans to build new data centres in San Antonio, Texas as well as a $900m facility alongside its Fort Meade, Maryland, headquarters. ®