Original URL: https://www.theregister.com/2012/06/29/surviving_it_crisis/
Do you work in IT at RBS? Or at the next place to get hit ...?
Here's a handy guide to riding out the storm
Posted in On-Prem, 29th June 2012 13:15 GMT
Those nice people at RBS have provided me with a worked example for how you can best get through fan-hitting situations without having to take up minicab driving or going to work in local government.
What I’m writing applies to the any of the very worst of screw-ups, when technology has gone so wrong that the IT director cancels his time at Wimbledon with his friendly vendors.
In these troubled times, even a badly paid dead-end job in RBS IT is something you want to keep hold of. Fixing the problem is at best 25 per cent of the management effort now being expended; the rest is finding scapegoats, shafting people whose job might be up for grabs and dodging retribution.
That leaves the IT pros, (ie, you, dear reader) exposed to angry powerful people protecting their arses at the IT staff's expense. RBS has had a policy of getting rid of experienced people who actually understand the obscenely complex ways of ancient multi-layered systems - otherwise known as shafting people over 30 to “save money”. Hence this may be your first time.
I draw heavily here on various things I have seen and yes, occasionally caused, in my 25-year stint in banking IT.
First, and most importantly, do not try blaming senior management out loud. You don’t get to be senior at RBS by knowing your job, be it due diligence in takeovers or understanding of technology. You get it by world-class skills in internal politics. That means that attacking them is like throwing a punch at a heavyweight boxer. You’ll both be embarrassed. You because you’re shown that your aggression dominates your judgment, the heavyweight because swatting you is beneath their dignity.
Visible Productivity
Do something visibly useful ASAP. The word “useful” is of course a cynical lie. I’m a City headhunter, so I’m a world-class cynic. what you need to do is quickly find a series of short, well-defined work items that you can complete and report to your boss both by email and at meetings. Validate some records, hand-check the syntax in batches, perform a series of tasks that can’t easily go wrong. That also includes working longer hours even if there’s nothing useful you can do. Visible effort and noticeable attitude is better for your career than competence.
Do not say you’ve fixed it
Some IT clusterfucks are single failure modes and are resolved by that one thing being fixed. The problem is that you can rarely know this for certain until the fix has been used for a while. The risk with both visible productivity and claiming success is that if you say “I’ve done X” and X turns out to be wrong, you’ve just given the political heavyweights a scapegoat – even if the original problem was nothing to do with you. (Yes, I’ve done this; learn from my mistakes.) You won’t be at the meetings where they decide that you’re the sacrificial virgin, so don’t expect to be able to defend yourself.
If it was your fault
Firstly, know that “fault” is a subjective term. The evidence from my esteemed colleague Anna Leach reports that you were given jobs to do that were within your reach but not your grasp.
Thus it is the fault of decisions-makers on their golf course, not yours.
Your lack of seniority works to your advantage. The big boys don’t really care about you personally. In fact firing you would be an admission that the decision to put the cheapest person they could find anywhere in the world in such a responsible position was a bad idea. Thus you are in the unique position in this process where telling the truth is in your best interests. For a given value of “truth”...
Actual lies will probably get you into more trouble, but you need to avoid admitting you were sloppy. Thus it is better to say “When I ran X, I believed it would work”, rather than “it was easier to run X”. It's also better to be frank up front, saying “the mistake I made was to do Y”, which draws a line under your position.
The longer management spend interrogating you, the greater the chance that someone will press the Eject button. I do occasional consultancy “interviewing” IT pros suspected of doing bad things, and I do get paid a few hundred quid an hour to do it. Banks only pay that if they are confident the interviewer will get the truth, so the first thing you need to do is sit down and get your story straight. Do not write it on your PC unless you have a death wish. Do offer to write a report on how it can be stopped from happening again. That will buy time and make you appear more professional than you look or feel. It is also an achievable goal.
Do not offer to resign, at least not until they make you an offer. You are not in your best condition to make career decisions and there is nothing to be gained in making them in the first week. If they do offer what senior management call an “elegant solution”, be aware that the standard is for them to pay for an independent lawyer to advise you and that your reference for future employment can be agreed and it is best for everyone to call it “redundancy”.
But that’s not the most likely outcome; they are more likely to move you to somewhere less important.
If it was someone who you manage
If you are an RBS manager, your first instinct will be to scapegoat the techie who pressed the wrong button, which is a mistake in a matter this large.
I don’t mean that ethically (remember what my job is), but practically. You need to talk of how you spotted the issue and how your team worked hard to fix it, citing names and specific achievements where possible. Your bosses won’t know the people or understand the issues, but it shows you are managing and that you are loyal to your troops.
In my experience senior management respect that and you need all the respect you can get right now. One line I have used to great effect when someone has a rant at a member of the team is to say clearly and just a tad louder than normal in front of both people: “Joe is part of the solution, if you’ve got a problem you shout at me, that’s my job: these guys are busting a gut to get us out of this mess.” That requires balls of course, as does anything else that gains respect.
Ideally you will have done things like this before so that the trust and support of your team was deeper and stronger. But it still can help and when you quit RBS to join a firm that values quality staff you’ll have learned things that make you more valuable.
Contrary to appearances not everyone at the top level of RBS is a narcissist who thinks only of golf and his bonus and they will be probing individual team members to work out who they can profitably blame. You need your people to be loyal, so be clear that you get the best credit indirectly, by praising your team, do not try and steal cred, it looks bad to your bosses and the biggest pothole that is waiting to swallow your career is if your team realise that you are trying to save yourself at their expense.
Reactions will vary from trying less hard to going over your head to explain their version of events to your bosses. Trust me, that won’t make you look good at all. You already know that you’re in the firing line, someone needs to be blamed and zapping you “saves money” as well as helping someone senior protect themselves. A couple of times I’ve been in charge of systems that I personally couldn’t help get back online, knowing just enough to get in the way.
If you’re a manager at RBS, odds are you are in that position, but that doesn’t mean you go home. Yes, I’ve seen managers do this, and the effort level of the team goes down hard. Order pizza. When staff can’t work any longer, order cabs to get them home. Get better coffee delivered than the crap they normally get.
Help your manager look good
Since the people making strategic decisions are in a different country, (if you’re in the UK part of RBS, a different planet) your group may live or die together since RBS management have a disdain for interacting with “grease monkeys” since so few play golf. That means they may well pick a bit of the org chart to delete.
The reason you need to supply your boss with positive and useful things he can say to his bosses is that his ability to defend you is wholly dependent upon this ammunition. Do not criticise him to other managers; not only does it undermine him in this fight but trust me when I say it will get back to him and he will want you out as soon as possible.
Senior staff may take you aside to get past his spin. Do not trust them. Their offers of protection have less credibility than the reports of a Greek finance minister. But you do need to play ball, and the advice I give to your boss applies here. Talk of how others are working hard and well and express tiredness but that you’re willing to see it through. One apple they may offer is promotion to take your boss’s position. Please don’t tell me you are naïve enough to believe this.
Sound relentlessly positive
Senior RBS managers will come to meetings that aren’t held at nice hotels near good golf courses for the first time in ages and they will try to work out who to blame and who should be promoted into the slots left vacant.
What to tell the Big Guys
If you work at RBS you understand that they don’t know or care anything about technology. They care about “business” – ie, accountancy and internal politics – and so they so will judge your competence on those grounds. Someone who says: “We shouldn’t have fired everyone who earned more than a Glasgow street cleaner” is digging his own grave at this sort of meeting and frankly it doesn’t help in any objective sense.
I’m not a CA-7 guru, so adjust these key phrases to your local environment. Good things to say at war meetings include:
“Let’s focus on delivering a solution, not on how we got here” – to be said every time someone who hasn’t read this piece complains.
“The focus needs to be on a simple solution that we know will work.”
“However confident we are that this will deliver, we should make sure we have a plan B.”
“I agree with you.”
“We need to work as a team on this.”
Do not get involved in arguments, wait until a consensus emerges and reinforce it.
What if someone wants to do something crazy?
With this sort of stress it is tempting to come down hard on people who suggest solutions that will not only corrupt the database but set a server on fire as well (yep, one of my guys did that once).
Firstly, you’re tired, so before you shout them down, think a bit longer. You may be putting a stop to a good solution being tried you may make yourself look bad, and yes in this sort of situation that is the priority. In fact you need to look like the one who is trying to build a rational consensus. Since RBS shafted its experienced people your manager may not have the depth to mediate when warring factions of IT pro demand radically different approaches.
Yes, I use the words “focus” and “delivery” too much. So should you.
If you’re a senior exec at RBS
You’re only reading this because someone printed it out for you and other media are quoting us. That’s because your IT staff trust random tech journalists that they’ve never met with the truth more than they trust you.
The stuff I’ve written for your lackeys isn’t actually new. Many will have worked out that telling the truth is bad for their careers so they will be optimistic to their managers who will be spinning to their bosses who will be delivering good news to theirs (repeat until your pay grade is reached). This will have catastrophic effects on your cushy job and if you want to keep your corporate golf membership and nice dinners with suppliers who take you to the Olympics and Wimbledon you need truth.
Look at what happened to Sir Ian Blair, head of Scotland Yard, when his officers shot a Brazillian electrician by mistake. Because his staff lied to him, he spouted crap on camera that made him look dishonest rather than incompetent. Choose which of those got him fired. You need to speak to foot soldiers alone and to use the political skills that got you there to get them to trust you. Because let’s be clear: a rational IT pro knows the only side you are on is yours, not the bank’s and certainly not theirs.
Ironically that means the peasants (don’t kid me you have a more polite word), might actually tell you the truth simply because they know that you’re too important to bother screwing with them personally and that when this is over, they won’t be worth persecuting.
Conversely, they won’t believe your offers of protection for dobbing in their managers. You didn’t get to where you are today because of your personal integrity. And yes we have met, at Bishopsgate as you may recall.
If you’re to keep the board happy, you need to present some plausible version of the truth together with a plan that actually stands a chance of stopping it happening again. Your PR people are earning their keep, stonewalling the press. That’s their job as you defined it to them. It’s what they always do. Is that working out well for you?
Blaming the Indians
They’re cheap and foreign, so Unite and other unions are quite happy for them to carry the can. It is tempting to join in. Apparently it’s fine for unions to characterise them as incompetent, but do not say this about the ones in your office because they have legal rights that Indians who live in India do not. Plus it is rude.
Accenture sell their “services” heavily on the idea that they can get the firm working together. Whatever you think of their delivery, no one anywhere is better at marketing services and they know that an article of faith for senior management in almost all firms is that if only “people worked as a team” everything would be great.
As an IT pro you will see this mess as proof that outsourcing to India is insane. But you’re not going to be on the golf course when the nice men point out to the decision makers that their bonuses will be improved by the short-term cost savings to be gained that way. They’ve learned from the masters at Accenture and will spin this mess as being a result of the two teams not being integrated closely enough, so expect the UK dog to be moved to the Indian tail.
Aftershocks
I wrote this piece because I’ve been involved in IT snafus myself (caused them, moi?) and my experience points to what may happen on 3/4 July. By this point the team dealing with the mess will be in a dreadful state. Bone weary, stressed and their stomachs retaliating from a diet of caffeine and fast food.
Some know that this morass was at least partly their fault and because no other UK bank was damned foolish enough to outsource to Hyderabad, using technology almost no one uses in India or the UK, they are looking at career meltdown.
The need to provide consistently positive messages to the golf players back in head office, meaning that there will have been pressure not to bring to light any risks, AKA “sending a consistent message”.
So you have people doing their job, but not thinking. Some critical staff will have gone home because they won’t be able to function even at the menial level required by RBS.
So I do expect an aftershock. Maybe a batch will be run twice (we’ve all done that – I’m just admitting it publicly), or test/debug code will be executed on the live database, or the “positive news” to senior management will have masked a problem that only gets worse until it gets fixed or RBS is shut down.
So I’m sticking my neck out and saying that during this week something else bad will happen. I put that at about 80 per cent during this week, with about a 40 to 50 per cent chance that the consequences will leak out of RBS, making the chances about one-in-three that customers suffer more pain.
That’s not enough for me to say that you shouldn't use RBS for any banking, but if it’s a transaction you care about, best do it somewhere else. ®
Before becoming a City headhunter, Dominic Connor was a mildly competent head of IT as well as developer for various banks. RBS still run some decade old code of his, he wishes them good luck with it.