Software upgrade KOs gas shipping auction system
Fax machines running red hot at UK's National Grid
Posted in Management, 26th October 2007 10:16 GMT
Free whitepaper – Hands on with Hyper-V 3.0 and virtual machine movement
A botched Oracle upgrade last Sunday has denied access to an online trading system used to sell space on the UK gas pipeline network. Fax machines have been running hot all week as every transaction has had to be entered manually.
The Gemini system at the centre of the affair doesn't control movements of gas, nor is it used to buy and sell gas. Rather, it is used to run daily auctions in which spare capacity on the pipeline network can be bought.
Gemini is run by a company called xoserve, jointly owned by the five companies which run the regional pipeline systems.
"Although xoserve itself is relatively new, many of our employees have a rich heritage of providing these types of services and running complex IS systems since the British gas industry was opened up entirely to supply competition in 1996," says the company - having only been founded in 2005.
This time, however, something went wrong. On Monday, National Grid informed its customers that: "Yesterday's Gemini Oracle upgrade was completed at 15:00, some three hours later than anticipated due to an issue with the software. This issue has been resolved. However, xoserve are experiencing ongoing operational issues..."
These operational issues have led to all external access to Gemini being cut off for the week, though data could still be input centrally. Hence, the cumbersome system of faxing, with National Grid in some instances having to fax back screenshots from Gemini for the customers' records.
xoserve staff declined to comment to the Reg, referring all questions to National Grid.
National Grid's Stuart Larque played down the seriousness of the situation. "Gemini is not physically connected to the infrastructure and it isn't used to buy or sell gas."
Larque said he couldn't comment on how the market in pipeline capacity might be affected, but did say that most capacity is bought and sold in large long-term deals unaffected by the current outage. The daily auctions now being conducted by fax handled only a relatively small part of the market.
Larque added that xoserve hoped to have access to Gemini restored later today. ®
Free whitepaper – Hands on with Hyper-V 3.0 and virtual machine movement
COMMENTS
Hhehehe
I used to work somewhere that did the pumping of gas and they had a lot to do with the trading as they needed to know how much to pump and to where.
They must be having a hell of a time...
On the down side when ever things like this happen the price of gas goes up.
@Mr. Beast
This is a production system where they were upgrading a system in place.
As I agree with your point that there should have been a parallel system in place, Dev, Test, UAT and production would have been a bit of overkill.
What is scary that they don't have a redundant system running in parallel that when they upgraded one machine, they could have failed over to the backup and either fixed the migration or reversed it.
But then again, they're running Oracle and not IBM's IDS (Informix). ;-)
Ahem...
"Unbreakable"...? Can we have an Ellison icon too, please?
business continuity plan?
Wonder if this xoserve mob bothered to think about putting one together? Seemingly not, otherwise it would have identified being able to rollback as a fallback when Oracle, M$ or whomsoever's 'upgrade' poos all over their heretofore stable system.
Thankfully, in this case, it seems the business is forward capacity planning rather than must-be-done-Now transactions.
Dev, Test, UAT and Production???
Whilst not all systems warrant it, most systems that are revenue generating, or client facing have, (or should have) an environment that mimics the production environment so that software patches or upgrades can be tested.
The creation of such environments however do tend to increase costs significantly, so you'll probably find that either there was no such system in place due to cost cutting measures or budget holders. Or the costs were such that the business decided that such outages were operationally acceptable when weighed up against how much an outage may cost and how much the test (or fail over) systems cost.
At least they had some BCP in place even if it relies on ancient technology (Who said that Fax was dead)
Enabling efficient data center monitoring
The new Office:
Top 10 SIEM implementer’s checklist
