Feeds

Bungled Hitachi SAN upgrade halts Oregon benefit payments

Salem witch hunt encounters, exorcises evil

Remote control for virtualized desktops

A bungled Hitachi storage area networking (SAN) upgrade has delayed the payout of over $18 million in benefits in Oregon.

The SAN failure began Monday evening in the state capital of Salem, and was fixed by Tuesday morning, with various state agencies taking the day to work through a backlog of jobs that had built up.

"Yesterday evening contractors from Hitachi were performing a scheduled upgrade at the state data center by adding hardware to the storage area network to give it more capacity," Matt Shelby, a spokesman for Oregon's Department of Administrative Services, told The Register.

"When they installed the new [capacity] things started to go south. It did not work as it was supposed to or intended to. They removed the board but by then the damage was done. We didn't lose any data, but it cut off communication lines between the SAN and the rest of the state."

The Hitachi SAN sits at the heart of the state's data center in Salem, which has functioned as the main data center for state services in Oregon since 2005. With its SAN out of action, agencies were not able to access important data.

This meant that $18 million worth of employment benefits were not paid out last night, the forestry service was not able to access maps, the state's job search portal iMatchSkills was down, and child support payments failed to go out on time, among other glitches big and small.

The problem was fixed by the early morning as on-site Hitachi engineers worked overnight coordinating with their headquarters in Japan to fix the problem, Shelby said.

"That downtime created quite the bottleneck. It took some time to do a couple of things. One, to bring [agencies] back online, and once agencies were back online there was some work that had piled back up at the agency levels."

Though he didn't describe it as a witch-hunt, Shelby indicated that now the problem has been dealt with state employees in Salem are putting together an incident report and have some questions for Hitachi. "Obviously, we want to discover how it happened," he said.

At the time of writing The Register had not received further information from state employees on the botched upgrade. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
NSA SOURCE CODE LEAK: Information slurp tools to appear online
Now you can run your own intelligence agency
Azure TITSUP caused by INFINITE LOOP
Fat fingered geo-block kept Aussies in the dark
Yahoo! blames! MONSTER! email! OUTAGE! on! CUT! CABLE! bungle!
Weekend woe for BT as telco struggles to restore service
Cloud unicorns are extinct so DiData cloud mess was YOUR fault
Applications need to be built to handle TITSUP incidents
Stop the IoT revolution! We need to figure out packet sizes first
Researchers test 802.15.4 and find we know nuh-think! about large scale sensor network ops
Turnbull should spare us all airline-magazine-grade cloud hype
Box-hugger is not a dirty word, Minister. Box-huggers make the cloud WORK
SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016
Flash WORM has a serious use for archived photos and videos
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
Microsoft adds video offering to Office 365. Oh NOES, you'll need Adobe Flash
Lovely presentations... but not on your Flash-hating mobe
prev story

Whitepapers

Free virtual appliance for wire data analytics
The ExtraHop Discovery Edition is a free virtual appliance will help you to discover the performance of your applications across the network, web, VDI, database, and storage tiers.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
10 threats to successful enterprise endpoint backup
10 threats to a successful backup including issues with BYOD, slow backups and ineffective security.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Website security in corporate America
Find out how you rank among other IT managers testing your website's vulnerabilities.