So, that vast IT disaster you may have caused? Come in, sit down
You won't be forced to dig your own grave
5 ways to prepare your advertising infrastructure for disaster
The RBS computer fiasco gives me an excuse to write about a sideline I have in interrogating IT professionals who are suspected of doing bad things.
Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most dangerous techie in your firm is not the disaffected sysadmin nor the under-performing developer but someone trying hard to do their job. The problems are being caused by him or her trying too hard.
Given my recent analysis of RBS's overnight transaction processing cock-up, it’s not likely they will ask me in to diagnose what happened, but this is what I’d do.
Ground rules
“Interrogation” is of course exactly the wrong word and if I’m dealing with a firm that has handled this sort of situation before, one that is utterly forbidden.
Often lawyers are involved by this point and they know that anything that can even be slightly represented as bullying can undermine the employer’s position big time and make a bad situation worse. Also if you're an employee in a position to cause the level of harm that justifies my high-but-excellent-value fees, losing you unnecessarily may well damage the business.
You will have political support either from the boss or colleagues and must be seen to be treated fairly. Of course I have no legal power to compel you to do anything. Since I’m far more polite in these meetings than I am in real life and not harassing you, any attempt to refuse to have a chat looks fatally bad.
I get to set some of the conditions, of which tea and biscuits are more important than some people realise. Ideally I’d be doing this in a bar, where the two of us relax, check each other out and get to an approximation of the truth that will allow a conclusion to be reached. That’s never going to happen, so the next best thing is to be a good host, preferably off-site with as few members of the firm’s management as I can engineer.
The exact nature of what happened often is an ingredient in a political spat. It’s not always clear to me what the client “wants” the answer to be, since it could well turn into ammunition in an internal fight. The killer is that this can be attached to a legal process where I may end up in court as a witness and no way am I lying under oath.
Please lie to me
It may seem odd to share some of my tricks with the almost 7 million people who read El Reg in any given month, given that I expect some of you to be involved. But they are integrity checks and I use “integrity” in both the ethical and database sense of the words.
I give the interviewee chances to lie to me. In my experience good techies are really uncomfortable with actually lying. They may skip details or exaggerate but very few can construct a consistent framework of untruths without the stress showing.
Trying to catch them out is the best fun I get with my clothes on: a multi-level puzzle with prizes as well as consequences. Either you know I’m a fellow geek because I’m nearly famous, or it comes as an unpleasant shock that the schmoozing City headhunter-type expresses “surprise” at the arcane details of your code or that you didn’t notice this particular flaw in the backup script.
You may be smarter than me - quite a lot of people are - but you have to be lucky all the time, and I need to catch you just once. You feeling lucky, punk?
A common defence, used by people who really ought to know better is that “it wasn’t me, it was someone using my ID”, which presumably sounds better in your head than out loud. In these days of CCTV, it rarely gives much protection. If you’re using that line, it’s hard not to believe you haven’t done something bad.
Lies make my job easier and make me look good to the people who have hired me. Let's be very clear here: they want a clear result. If I catch you in a lie then to the best of my ability I won’t react, but I recalibrate the other things you tell me in a very different light.
A provable lie means your bosses get that clear result. Your political support will not only vanish but your supporters will feel betrayed and turn on you in a way that rabid wolves would regard as harsh.
Preparation
I don’t do forensics. Yes, I can do sector-level drive scans and know more about SQL server logs than is good for me, but if you are suspected of being very naughty I will refuse point blank to even touch your PC. Instead, I'll call up someone like Guidance Software to provide an evidence trail that can be used in court. Though, it usually doesn’t go that far.
I do need to look at what’s been going on, which does cause considerable discomfort since that means looking at the rather less polished parts of the operation - and is why I have serious non-disclosure agreements and no real specifics are in this article.
Why me?
I sometimes ask that myself. I certainly get a feeling of “there but for the grace of small gods go I” during this process. If I was a skeleton with a scythe I’d probably get a warmer reception from the rest of the team, who know any number of people in their firm who can evaluate what you did better than I can.
The problem is that they are conflicted and may be implicated themselves. A good boss will defend his staff, and your colleagues know that “shortcuts” are necessary to meet deadlines and resource constraints - as well as the standard defence of “everyone else was doing it” - which is both crap and true. Someone may want to get you out of a petty personality squabble that you see in any organisation, and it may be that someone is trying to shift blame. In total there’s not likely to be anyone inside the firm that can be seen to be entirely objective, so I get a call.
