Feeds

Bungled Hitachi SAN upgrade halts Oregon benefit payments

Salem witch hunt encounters, exorcises evil

Top three mobile application threats

A bungled Hitachi storage area networking (SAN) upgrade has delayed the payout of over $18 million in benefits in Oregon.

The SAN failure began Monday evening in the state capital of Salem, and was fixed by Tuesday morning, with various state agencies taking the day to work through a backlog of jobs that had built up.

"Yesterday evening contractors from Hitachi were performing a scheduled upgrade at the state data center by adding hardware to the storage area network to give it more capacity," Matt Shelby, a spokesman for Oregon's Department of Administrative Services, told The Register.

"When they installed the new [capacity] things started to go south. It did not work as it was supposed to or intended to. They removed the board but by then the damage was done. We didn't lose any data, but it cut off communication lines between the SAN and the rest of the state."

The Hitachi SAN sits at the heart of the state's data center in Salem, which has functioned as the main data center for state services in Oregon since 2005. With its SAN out of action, agencies were not able to access important data.

This meant that $18 million worth of employment benefits were not paid out last night, the forestry service was not able to access maps, the state's job search portal iMatchSkills was down, and child support payments failed to go out on time, among other glitches big and small.

The problem was fixed by the early morning as on-site Hitachi engineers worked overnight coordinating with their headquarters in Japan to fix the problem, Shelby said.

"That downtime created quite the bottleneck. It took some time to do a couple of things. One, to bring [agencies] back online, and once agencies were back online there was some work that had piled back up at the agency levels."

Though he didn't describe it as a witch-hunt, Shelby indicated that now the problem has been dealt with state employees in Salem are putting together an incident report and have some questions for Hitachi. "Obviously, we want to discover how it happened," he said.

At the time of writing The Register had not received further information from state employees on the botched upgrade. ®

High performance access to file storage

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
BOFH: Oh DO tell us what you think. *CLICK*
$%%&amp Oh dear, we've been cut *CLICK* Well hello *CLICK* You're breaking up...
Just what could be inside Dropbox's new 'Home For Life'?
Biz apps, messaging, photos, email, more storage – sorry, did you think there would be cake?
IT bods: How long does it take YOU to train up on new tech?
I'll leave my arrays to do the hard work, if you don't mind
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.