'Inexperienced' RBS tech operative's blunder led to banking meltdown
Hyderabad job ad shows outsourcing in CA-7 team
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
Exclusive A serious error committed by an "inexperienced operative" caused the IT meltdown which crippled the RBS banks last week, a source familiar with the matter has told The Register. Job adverts show that at least some of the team responsible for the blunder were recruited earlier this year in India following IT job cuts at RBS in the UK.
Following our revelation yesterday that a bungled update to CA-7 batch processing software used by RBS lay behind the collapse, further details have emerged. According to a Register source who worked at RBS for several years, an inexperienced operative made a major error while performing the relatively routine task of backing out of an upgrade to the CA-7 tool. It is normal to find that a software update has caused a problem; IT staff expect to back out in such cases.
But in the process of backing out a major blunder was committed, according to our source. It was this error which made the task of restoring services so prolonged:
When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling.
That created a large backlog as all the wiped information had to be re-inputted to the system and reprocessed. A complicated legacy mainframe system at RBS and a team inexperienced in its quirks made the problem harder to fix, our source adds.
CA Technologies - the makers of the CA-7 software at the heart of the snarl-up - are helping RBS to fix the disaster that has affected 16.9 million UK bank accounts.
"RBS is a valued CA Technologies customer, we are offering all assistance possible to help them resolve their technical issues," a spokeswoman told The Register.
CA added that RBS's technical issues were "highly unique to their environment".
The batch scheduling tool CA-7 is widely used and generally considered to be very reliable, so it appears that the error - that meant millions of accounts have registered incorrect balances for many days - sprang from the oversight of the technology at RBS.
RBS: "No evidence" this is connected to outsourcing
The CEO of RBS Group, Stephen Hester, has said that there is no evidence that the problem is connected to lack of investment in technology at RBS and the outsourcing of IT jobs to India. Asked directly by Sky news yesterday, he said:
Well I have no evidence of that. The IT centre - our main centre, we’re standing outside here in Edinburgh, [is] nothing to do with overseas. Our UK backbone has seen substantial investment.
We put our sources' revelations regarding poor levels of knowledge in the handling of CA-7 to RBS spokespeople - and reminded the company's representatives of the job ad showing that at least some of the firm's CA-7 handlers were recruited in India this year, as we reported yesterday. We offered the company an opportunity to confirm that the critical blunder was committed by a UK-based rather than an India-based operator.
However the bank's spokesmen refused to offer any further comment. ®
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
COMMENTS
Re: no backup of the schedule?
To answer Lee Dowling questions:_
Q1. How did someone inexperienced get on the team?
A1. RBS got rid of all the experienced staff and outsourced their jobs
Q2. How did they get access to the schedule controls?
A2. RBS got rid of all the experienced staff and outsourced their jobs
Q3. How did they managed to delete EVERYTHING on there?
A3. RBS got rid of all the experienced staff and outsourced their jobs
Q4. Why did the software allow such deletion without confirmation?
A4. Computers do what you tell them, which more often than not is not what you want them to do, that's why you get experienced staff to do it. You can try this yourself, open a command prompt on you PC and enter "format c:" and answer "Y" to the question.
Q5. Why is there not a rollback or even versioning function for the schedules?
A5. There is but RBS got rid of all the experienced staff who knew how to do it and outsourced their jobs
Q6. Why, precisely, does one mess-up by one employee in front of one computer put your ENTIRE BANKING SYSTEM out of action, nationwide?
A6. RBS got rid of all the experienced staff and outsourced their jobs
Simples
RBS: "No evidence" this is connected to outsourcing
Well of course you wont find evidence if you don't go out looking for it!
This is worse than incompetence, this is 'bury your head in the sand' wilful ignorance. They made experience staff redundant, and when they gave those responsibilities to someone with little experience of the system, they cocked it up.
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
Steps to Take Before Choosing a Business Continuity Partner
Enabling efficient data center monitoring
