Original URL: https://www.theregister.com/2012/07/03/forensic_it_interviews/
So, that vast IT disaster you may have caused? Come in, sit down
You won't be forced to dig your own grave
Posted in On-Prem, 3rd July 2012 12:00 GMT
I have a little sideline in interrogating IT professionals who are suspected of doing bad things.
Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most dangerous techie in your firm is not the disaffected sysadmin nor the under-performing developer but someone trying hard to do their job. The problems are being caused by him or her trying too hard.
Given my analysis of RBS's overnight transaction processing cock-up, it’s not likely they will ask me in to diagnose what happened, but this is what I’d do.
Ground rules
“Interrogation” is of course exactly the wrong word and if I’m dealing with a firm that has handled this sort of situation before, one that is utterly forbidden.
Often lawyers are involved by this point and they know that anything that can even be slightly represented as bullying can undermine the employer’s position big time and make a bad situation worse. Also if you're an employee in a position to cause the level of harm that justifies my high-but-excellent-value fees, losing you unnecessarily may well damage the business.
You will have political support either from the boss or colleagues and must be seen to be treated fairly. Of course I have no legal power to compel you to do anything. Since I’m far more polite in these meetings than I am in real life and not harassing you, any attempt to refuse to have a chat looks fatally bad.
I get to set some of the conditions, of which tea and biscuits are more important than some people realise. Ideally I’d be doing this in a bar, where the two of us relax, check each other out and get to an approximation of the truth that will allow a conclusion to be reached. That’s never going to happen, so the next best thing is to be a good host, preferably off-site with as few members of the firm’s management as I can engineer.
The exact nature of what happened often is an ingredient in a political spat. It’s not always clear to me what the client “wants” the answer to be, since it could well turn into ammunition in an internal fight. The killer is that this can be attached to a legal process where I may end up in court as a witness and no way am I lying under oath.
Please lie to me
It may seem odd to share some of my tricks with the almost 7 million people who read El Reg in any given month, given that I expect some of you to be involved. But they are integrity checks and I use “integrity” in both the ethical and database sense of the words.
I give the interviewee chances to lie to me. In my experience good techies are really uncomfortable with actually lying. They may skip details or exaggerate but very few can construct a consistent framework of untruths without the stress showing.
Trying to catch them out is the best fun I get with my clothes on: a multi-level puzzle with prizes as well as consequences. Either you know I’m a fellow geek because I’m nearly famous, or it comes as an unpleasant shock that the schmoozing City headhunter-type expresses “surprise” at the arcane details of your code or that you didn’t notice this particular flaw in the backup script.
You may be smarter than me - quite a lot of people are - but you have to be lucky all the time, and I need to catch you just once. You feeling lucky, punk?
A common defence, used by people who really ought to know better is that “it wasn’t me, it was someone using my ID”, which presumably sounds better in your head than out loud. In these days of CCTV, it rarely gives much protection. If you’re using that line, it’s hard not to believe you haven’t done something bad.
Lies make my job easier and make me look good to the people who have hired me. Let's be very clear here: they want a clear result. If I catch you in a lie then to the best of my ability I won’t react, but I recalibrate the other things you tell me in a very different light.
A provable lie means your bosses get that clear result. Your political support will not only vanish but your supporters will feel betrayed and turn on you in a way that rabid wolves would regard as harsh.
Preparation
I don’t do forensics. Yes, I can do sector-level drive scans and know more about SQL server logs than is good for me, but if you are suspected of being very naughty I will refuse point blank to even touch your PC. Instead, I'll call up someone like Guidance Software to provide an evidence trail that can be used in court. Though, it usually doesn’t go that far.
I do need to look at what’s been going on, which does cause considerable discomfort since that means looking at the rather less polished parts of the operation - and is why I have serious non-disclosure agreements and no real specifics are in this article.
Why me?
I sometimes ask that myself. I certainly get a feeling of “there but for the grace of small gods go I” during this process. If I was a skeleton with a scythe I’d probably get a warmer reception from the rest of the team, who know any number of people in their firm who can evaluate what you did better than I can.
The problem is that they are conflicted and may be implicated themselves. A good boss will defend his staff, and your colleagues know that “shortcuts” are necessary to meet deadlines and resource constraints - as well as the standard defence of “everyone else was doing it” - which is both crap and true. Someone may want to get you out of a petty personality squabble that you see in any organisation, and it may be that someone is trying to shift blame. In total there’s not likely to be anyone inside the firm that can be seen to be entirely objective, so I get a call.
Judgement
One ex-colleague of mine was fired for putting a screwdriver violently through his system before security took him away, but in reality it’s vanishingly rare that I can say with 100 per cent confidence that you’ve been naughty or nice.
This sometimes very much disappoints a client because they have to make binary decisions, which is tough when the so-called expert writes a three-page report that says in effect: “85 per cent likely it was the sort of blunder any IT pro makes once per career, 10 per cent he is out of his depth, and 5 per cent it happened on purpose”. If you want someone to write 100 per cent on any of those numbers you need someone either much smarter or less honest than me.
Programming is of course the most contentious area, because regardless of what it says in the corporate development standards all major programs contain snippets downloaded from the web, code from previous employers and bits of debug that should have been removed years ago but every time someone tried the system started producing frightening output.
I even occasionally note with grim satisfaction that they have acquired some of my published code. Even with a decent version control system it is hard to say that a given person wrote a particular fragment of code, and you ought to be more surprised than you are how many teams don’t have a good VCS.
That means the interview may have a “why did you write it that way?” section which requires the full limit of my programming experience to follow. Not just obtuse C++ or single letter variable VB but getting through the “yeah, that was a dirty hack, but to get it done on time we called all the other error conditions 'insufficient memory'" quips.
Obviously I can’t talk about my clients, but for various nearly good reasons I got to read the Windows source code and found a lot of those, so don’t for a minute believe that sort of tomfoolery isn’t standard, so that’s what I report. I’m a realist who would actually be suspicious of very clean code.
Reporting back
One thing that non-IT types find hard to grasp is that the scale of the consequences and the cause don’t correlate at all - although RBS management probably gets that idea now.
It’s not my job to say fire or keep you, not because I’m afraid of sticking my neck out (see my other articles), but because very rarely am I given the whole picture, sometimes for good reasons and sometimes not. There’s also a legal issue in that the decision to fire you needs to be taken by your employer as part of a proper process. It is not in their interests to say they delegated it to an outsider.
If you find yourself sitting next to me... you need to stop digging. There is already a hole and lying will just cause my smile to become a little more fixed as I hand you the spade.
Answer the questions, feel free to explain things you think I haven’t understood or been told, but be aware that the more times I have to ask a question the less likely I am to believe the answer. I’m paid for my time and thus quite happy to do a Paxman on you. Possibly the most valuable thing you will get from this is knowing that if you are in the UK or EU and this stuff gets used against you then you have a right to a copy of my report. Advice that I hope you will find is completely useless to you. ®
Dominic Connor used to boss around gangs of quants, developers and sysadmins before moving to a life of headhunting in the City spiced up with consultancy.