Feeds

Bungled Hitachi SAN upgrade halts Oregon benefit payments

Salem witch hunt encounters, exorcises evil

Providing a secure and efficient Helpdesk

A bungled Hitachi storage area networking (SAN) upgrade has delayed the payout of over $18 million in benefits in Oregon.

The SAN failure began Monday evening in the state capital of Salem, and was fixed by Tuesday morning, with various state agencies taking the day to work through a backlog of jobs that had built up.

"Yesterday evening contractors from Hitachi were performing a scheduled upgrade at the state data center by adding hardware to the storage area network to give it more capacity," Matt Shelby, a spokesman for Oregon's Department of Administrative Services, told The Register.

"When they installed the new [capacity] things started to go south. It did not work as it was supposed to or intended to. They removed the board but by then the damage was done. We didn't lose any data, but it cut off communication lines between the SAN and the rest of the state."

The Hitachi SAN sits at the heart of the state's data center in Salem, which has functioned as the main data center for state services in Oregon since 2005. With its SAN out of action, agencies were not able to access important data.

This meant that $18 million worth of employment benefits were not paid out last night, the forestry service was not able to access maps, the state's job search portal iMatchSkills was down, and child support payments failed to go out on time, among other glitches big and small.

The problem was fixed by the early morning as on-site Hitachi engineers worked overnight coordinating with their headquarters in Japan to fix the problem, Shelby said.

"That downtime created quite the bottleneck. It took some time to do a couple of things. One, to bring [agencies] back online, and once agencies were back online there was some work that had piled back up at the agency levels."

Though he didn't describe it as a witch-hunt, Shelby indicated that now the problem has been dealt with state employees in Salem are putting together an incident report and have some questions for Hitachi. "Obviously, we want to discover how it happened," he said.

At the time of writing The Register had not received further information from state employees on the botched upgrade. ®

Providing a secure and efficient Helpdesk

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.