Feeds

RBS collapse details revealed: Arrow points to defective part

Software that caused cockup apparently run from India

Boost IT visibility and business value

Exclusive The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register.

And at least some of the support staff for that software have been outsourced to India - as recently as February.

Batch scheduling software is used to process routine jobs on a computer without the need for manual input: jobs are prioritised, scheduled and performed automatically - saving human time and using computer resources more efficiently. However the batch scheduling software needs to be maintained and overseen and it's there that our two sources believe the error occurred.

The main batch scheduling software used by RBS is CA-7, said one source, a former RBS employee who left the company recently.

They have more than one type of batch processing software. But the vast majority is run on CA-7.

A second former employee, familiar with RBS's use of the CA-7 tool, confirmed that the majority of batch processing at RBS was performed using the software and fingered a failed update as the cause of the problem:

RBS do use CA-7 and do update all accounts overnight on a mainframe via thousands of batch jobs scheduled by CA-7 ... Backing out of a failed update to CA-7 really ought to have been a trivial matter for experienced operations and systems programming staff, especially if they knew that an update had been made. That this was not the case tends to imply that the criticisms of the policy to "off-shore" also hold some water.

RBS in previous statements have confirmed that the problem occurred on Tuesday night. RBS engineering teams started to work on a fix on Wednesday morning, but the problem was not cracked until the weekend.

Possible reasons for the failure of the update include inadequate testing, perhaps exacerbated by poor documentation.

Staff who oversee batch scheduling for RBS are based in India

At least some staff overseeing CA-7 batch scheduling software used by RBS are based in India as this job advert for a CA-7 admin in Hyderabad, dating from February this year shows:

A job advert for a CA-7 post for RBS, screengrab Quikr

The job advert, dating from Feb 15th is marked as an "Urgent Requirement"

Note that this is an Urgent Requirement from RBS. The job would be paid 8-10 lakh rupees, a salary of roughly £9,000 - £11,000.

RBS have shifted IT jobs to India as the bank has tried to cut costs after its bailout by the government in 2008. There have been significant job cuts at the RBS group in the past 3 years. The government still owns the majority of RBS.

RBS's response

When asked to confirm that the CA-7 software lay behind the problems seen last week, RBS refused to confirm what software exactly was involved.

The RBS spokesman stated that "the software error occurred on a UK-based piece of software" but declined to give details on where the staff overseeing the software were based. ®

Build a business case: developing custom apps

More from The Register

next story
Kate Bush: Don't make me HAVE CONTACT with your iPHONE
Can't face sea of wobbling fondle implements. What happened to lighters, eh?
Video of US journalist 'beheading' pulled from social media
Yanked footage featured British-accented attacker and US journo James Foley
Caught red-handed: UK cops, PCSOs, specials behaving badly… on social media
No Mr Fuzz, don't ask a crime victim to be your pal on Facebook
Ballmer leaves Microsoft board to spend more time with his b-balls
From Clippy to Clippers: Hi, I see you're running an NBA team now ...
Online tat bazaar eBay coughs to YET ANOTHER outage
Web-based flea market struck dumb by size and scale of fail
Amazon takes swipe at PayPal, Square with card reader for mobes
Etailer plans to undercut rivals with low transaction fee offer
Assange™: Hey world, I'M STILL HERE, ignore that Snowden guy
Press conference: ME ME ME ME ME ME ME (cont'd pg 94)
Call of Duty daddy considers launching own movie studio
Activision Blizzard might like quality control of a CoD film
US regulators OK sale of IBM's x86 server biz to Lenovo
Now all that remains is for gov't offices to ban the boxes
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.