'Mainframe blowout' knackered millions of RBS, NatWest accounts
Bankers blame hardware fault, sources point to IBM big iron
A hardware fault in one of the Royal Bank of Scotland Group's mainframes prevented millions of customers from accessing their accounts last night.
A spokesman said an unspecified system failure was to blame after folks were unable to log into online banking, use cash machines or make payments at the tills for three hours on Wednesday evening. The fault affected customers of RBS, NatWest and Ulster Bank.
The taxpayer-owned financial giant stressed this was not repeat of the catastrophic three-day outage that kicked off less than a year ago, during which RBS banks couldn't process payments to millions of accounts. The cock-up left infuriated customers short of cash and unable to settle bills, shop for food or keep up with mortgage repayments.
Speaking of yesterday's titsup service, an RBS Group spokesman told The Reg: "This problem was caused by a hardware fault and was not related to the issues we experienced last summer.”
El Reg traced the cause of last year’s downtime to human error: an inexperienced IT operative hit the wrong button during what should have been a routine overnight CA-7 batch job to process inbound payments. The cock-up caused a huge backlog that took days, and in some cases weeks, to clear.
Our sources close to the banking group's IT systems told us today that last night's outage was likely a failure on the IBM mainframe that handles customer accounts. This fault may have been something as simple as a corrupted hard drive, broken disk controller or interconnecting hardware.
But such a minor issue shouldn't have taken down core activities. In theory, the banking group’s disaster-recovery procedures should have kicked in straight away without a glitch in critical services.
Yet, between 9pm and 11pm, customers were unable to access accounts online, by phone or through cash machines and smartphones.
One source told The Reg: “This one looks like a standard outage, rather than anything more complicated. It's just so unusual for these to happen with mainframes.”
RBS Group runs its core banking operations, including all customer accounts, on IBM zSeries machines, among the most reliable hardware in the industry.
Another source told The Reg that human error is “most likely” for the actual delay in initiating the recovery; the bank’s IT procedures will in some way require system administrators to understand a problem before they start flipping switches.
If you're in the know and would like to add to the coverage of this latest outage at RBS Group, drop The Reg a note or call the London office on 020 3189 4620. ®