Bungled Hitachi SAN upgrade halts Oregon benefit payments
Salem witch hunt encounters, exorcises evil
A bungled Hitachi storage area networking (SAN) upgrade has delayed the payout of over $18 million in benefits in Oregon.
The SAN failure began Monday evening in the state capital of Salem, and was fixed by Tuesday morning, with various state agencies taking the day to work through a backlog of jobs that had built up.
"Yesterday evening contractors from Hitachi were performing a scheduled upgrade at the state data center by adding hardware to the storage area network to give it more capacity," Matt Shelby, a spokesman for Oregon's Department of Administrative Services, told The Register.
"When they installed the new [capacity] things started to go south. It did not work as it was supposed to or intended to. They removed the board but by then the damage was done. We didn't lose any data, but it cut off communication lines between the SAN and the rest of the state."
The Hitachi SAN sits at the heart of the state's data center in Salem, which has functioned as the main data center for state services in Oregon since 2005. With its SAN out of action, agencies were not able to access important data.
This meant that $18 million worth of employment benefits were not paid out last night, the forestry service was not able to access maps, the state's job search portal iMatchSkills was down, and child support payments failed to go out on time, among other glitches big and small.
The problem was fixed by the early morning as on-site Hitachi engineers worked overnight coordinating with their headquarters in Japan to fix the problem, Shelby said.
"That downtime created quite the bottleneck. It took some time to do a couple of things. One, to bring [agencies] back online, and once agencies were back online there was some work that had piled back up at the agency levels."
Though he didn't describe it as a witch-hunt, Shelby indicated that now the problem has been dealt with state employees in Salem are putting together an incident report and have some questions for Hitachi. "Obviously, we want to discover how it happened," he said.
At the time of writing The Register had not received further information from state employees on the botched upgrade. ®