Having bank problems? I feel bad for you son: I've got 25 million problems, but a bulk upload ain't one
It was acceptable in the Eighties
Who, Me? Sunday is gone and Monday is here. To ring in the week, please join us in welcoming the latest addition to the shedload of shame that is The Register's Who, Me? column.
Today's toe-curler comes from a reader we shall refer to as "Geoff" who, as a newly qualified COBOL and Assembly-language programmer, was working for what he told us was "the biggest bank in the country."
British account holders with a nervous disposition need not look away. Geoff assured us "this was NOT in the UK".
Many decades ago, when the story begins, Geoff was "testing the ability of the bank's ageing mainframe system to handle and post a bulk upload of customer debits".
Stuff like cheques and credit-card slips – that sort of thing.
The hope was to blast through the lot in one go "rather than the drip-feed that was the prevailing method".
To check things out, Geoff needed some test data. Those being very different times, he "took the previous day's transactions for the whole bank, formatted them as per the bulk-load requirement, and saved the file".
After all, you can't beat real-world data, can you?
And this, as Geoff explained, was as real world as it got: "This represented about 25 million transactions with a total value in the billions, quite literally."
Ever the cautious fellow, Geoff called up the Computer Centre Manager and asked if the upload service could be switched to the test environment for 10 minutes so the test could be run.
Upon getting the affirmative, Geoff readied himself. At the last moment he noticed an error in his data, so paused to fix it.
Whew, that was a close one.
Geoff carried on, and "hit Send. It worked beautifully, for once the 56kbps modem we were using behaved itself, and the whole upload went perfectly smoothly."
Hurrah! Flushed with satisfaction at a job well done, he called the CCM again to let him know that the upload service could be switched back to live.
"Oh, don't worry, I switched it back already. I waited 10 minutes like you said."
Geoff told us: "More than 30 years later I can still vividly remember the black cloud of despair descending on me.
"He'd switched it back to Live while I was busy doing that edit.
"Every single transaction was posted in near-real time to real customer accounts, duplicating the entire previous day's debit run. Every customer got debited twice."
Register readers are clearly a responsible lot, and Geoff confessed all to his boss. He was hauled before an emergency board meeting, which included the chairman of the Bank, to explain what he had done and, more importantly, how he was going to fix it.
The story even made the front page of the newspapers ("thankfully without my name attached").
As for fixing the thing: "I did in fact fix it, by manipulating the same file into a collection of 'reversals', turning debits into credits, and uploading that. It cancelled out my catastrophic error (someone else took care of the interest calculations!)"
However, it took two days and nights of pizza and caffeine-fuelled panic to deal with the fallout.
Geoff observed: "Unsurprisingly, certain security protocols were put in place as a result, not least the requirement always to anonymise test data!"
Indeed. We put the story to a friendly engineer working in the UK's financial sector who, after the laughter subsided, told us that "the regulator would be very 'interested'" should such a thing happen these days, the bank's reputation would be left in "tatters", and the use of live data? Ouch.
Still, he remarked that Geoff had done the right thing by confessing.
As for his fate?
"I didn't get fired (I even got promoted a year later)."
Honesty, it seems, really does pay.
Ever run roughshod over regulations and nearly broken a bank? Who, Me? needs to hear from you. Not least so we know where not to leave our meagre savings. ®
Sponsored: What next after Netezza?