CIOs, what does your nightmare before Christmas look like?

Profitable bugs

Many problems are caused by lack of proper stress testing and can take literally years to emerge as a result. Good stress testing always breaks the system because you’re finding that point. As one firm found, this can be rather profitable.

They got a “give us the money now or we turn you off” death threat from a critical supplier, who hadn’t been paid for months. Even when they saw the bug, they found it hard to believe that when their bills payable system hit 999 it simply threw all the other bills away, but it was so. With a flow of around 2,000 bills per month it meant that on average they were only paying half what they intended to.

Of course, it was a lottery where one supplier had been repeatedly unlucky. This was both good and bad, since the firm discovered it was much less cash rich than it expected, but good because the treasury function had invested wisely and made a nice profit on the cash it hadn’t paid out.

A less profitable case was a firm which made very good money on premium rate text messages. One genius hit upon a viral messaging stategy for Valentine's Day, where the target of your lust could be sent a message and they’d be encouraged to send it to their attractors. Our CTO had been dragged out of bed by a panicked office because the damned thing worked.

Far, far too well.

The system was set up for 40-50,000 messages per day, which at a pound per message was good business. The queue at that point was heading up towards millions and the system simply couldn’t cope, requiring continuous profit-destroying reboots. The firm made only a tiny fraction of what could have been sucked out of the pockets of gullible fools.

A similar problem was a firm which quite innocently sent a marketing message to a small number of customers, but made the classic blunder of using CC rather than BCC. Sadly there were a couple of buggy Windows Small Business Servers in the list which apparently had ambitions to become Big Business Servers and went into a loop spamming the messages repeatedly and exponentially – until the original sender had been deeply blacklisted.

Expanding a data centre can be viciously expensive and often not possible in the current building, so many CTOs are one hot summer away from a meltdown – or in the case of one ISP, over the edge of what the aircon could cope with. This got ugly when a big ISP’s Brick Lane centre got so hot the BOFHs had to strip down to their underwear, cool down and rush back in, just as a BBC film crew turned up to film the disaster.

Risk register

Writing up the Round Tables, I refer to our members as CTOs, CIOs, IT execs etc, but often the job is simply managing the threat register, which we learned varies a lot at different firms. They’re a bit cynical about the standard risk register (no that’s not part of el Reg) because mostly it is a once a year bit of bureaucracy to keep the auditors happy.

Actually its role is to make them unhappy because the point of audit is to look for work to enrich the consultancy arm of the auditors. It needs to be a living document of the business risks you are managing and these days the boundary is both fuzzy and political. It isn’t enough to highlight a risk; you need to find someone with a budget to take ownership of the problem and that is rarely easy.

Saying “I told you about this risk a year ago” is a weak defence when the risk becomes a reality. As Chief Threat Officers we pondered about how we present Black Swans of low risk high consequence events and of course the infamous, known unknowns and unknown unknowns. One tip is to call them Black Swans, even if they’re not, simply because the more pretentious end of MBA courses now talk about them without actually evincing any understanding of probability, so you can bluff it.

Now that so many systems face an outside world full of criminals, some of our IT execs now manage ‘fraud queues’ where managing the various attempts to either steal, or buy services that enable them to steal, are so common it has become as routine as shoplifting in retail, where it’s just another business process. They were all remarkably relaxed about cybercrime, not because of any attitude that it “couldn’t happen to me” but mostly because it had and they had gotten through it intact.

Kidnap as a management strategy

One thing made worse by the cloud is that vendors have even less incentive to play nicely together. Those IT execs who solved “it wasn’t me it was him” squabbles by simply saying “no one leaves until this is fixed” are stymied when the cloud rep (wherever his office is) seems to be perpetually in a meeting and emails are met with “Thank you for your message, all messages are important to us and we will get back to you just as soon as we can”. As Office 365 customers have found, there is little come back when, as their PR people put it, “a few customers temporarily experienced some problems”.

It may not shock you to learn that no problem in the history of the world has ever been described as affecting a lot of customers.