Magic Quadrant for Enterprise Backup/Recovery

Exclusive Cost-cutting RBS management had halved the team within which the banking group's recent data disaster happened, sources have told The Register. The sacked British employees were replaced by staff in India, and there had been concerns about the quality of the work done in India for a lengthy period prior to last week's catastrophe.

Mishandling of batch schedule data while backing out of an update to CA-7 batch processing software last week caused the disruption that led to 16.9 million customers at RBS, Natwest and Ulsterbank being frozen out of their accounts for days, and ongoing issues in some cases.

The actual CA-7 software support team is wholly based in the UK and according to our sources, RBS has not cut that team.

However the batch scheduling team in the UK was cut, and certain of the jobs taken from the UK were made up with staff from RBS's Indian offices at Technology Services India. Though there were competent people working there, our sources said quality of work from India was patchy and that they had raised these problems with RBS management for the past two years.

Batch scheduling and why it's important

Batch scheduling is intrinsic to the process of the nightly data crunching performed by CA-7. The batch scheduling team prepare and schedule data for input into CA-7, gathering it from RBS's systems.

One source said:

The batch team was about 60 guys ... they ran applications that chose the jobs that ran - maintaining the CA-7 schedule, not the CA-7 support ... There were no redundancies in the CA-7 team but the batch support were taken down to about 30.

A second source told us that all UK-based RBS IT teams in areas considered non-critical had suffered redundancies of 50-70 per cent, depending on the individual teams. In many cases the headcount in the cut departments was maintained by hiring staff in RBS's Technology Services India.

The job advert we have previously reported for a CA-7 consultant in India would have been for the batch scheduling team, not for the CA-7 software support team itself. These screen grabs of a CV from LinkedIn, supplied to us by the cantankerous blog (the information has now been taken down) confirm that RBS runs at least one batch scheduling team in its offshore branch in India.

This meant RBS lost experienced employees familiar with their complex mainframe systems:

On the batch team a lot of them were lifers, had been working on that for many years.

As noted in previous Reg stories, it is understood that the error was made when backing out of an upgrade from CA-7 v11.1 to v11.3. The CA-7 upgrade took place at the weekend of 16/17th June and a problem was noticed on Monday which prompted a back-out from the upgrade on Tuesday night. In the back-out, an "inexperienced operator" made the wrong move and the day's data was wiped from the system. This created the backlog that has taken so long to clear.

Quality problems with work from RBS India

Both sources told us that though RBS's Technology Services India branch contained very competent people, the quality of work from there was patchy, and team managers frequently flagged up problems from about the quality of the work from India.

In several cases it was hard to recruit enough staff with the right skills.

"Team managers were struggling to get enough qualified staff, and were forced to take on people they had previously rejected. They were forced to take them to keep headcount," one source says. "People were considered to be fine technically but inexperienced."

A second source backed that up: "They obviously learn UNIX at uni but they don't know about IBM mainframes."

As a result there seemed to be frequent issues with the work performed. "We experienced great frustration" said one source, "some teams were great, but many we found we couldn't trust or struggled with."

One of our sources described an instance where he had overseen an upgrade to one of the bank's important systems (not the batch processing in this case) and described how the whole team involved in RBS UK and RBS India had to collaborate on the upgrade plan and the back-out plan:

I sent an email about the back-out plan [to the team in Technology Services India] and had to send it to them three times. All they had to do was copy and paste something into the back-out plan. In the end I had to get the quality control guy to cut and paste this into the back-out plan document. It took three emails just to copy and paste something.

The same source said that he was disappointed, because he been loyal to the company for years and felt that this mistake was very avoidable:

"It could have been prevented if the management had listened to us."

Asked for comment, RBS supplied The Register with this statement:

We have been clear we will fully investigate the causes of this incident. We hope people will understand that right now our complete focus is on fixing this problem and helping our customers.

The management and execution of batch processing is carried out in Edinburgh as has been the effort to recover and resolve this issue.

®

What you need to know about cloud backup