Christmas Eve ERP migration derailed by silly spreadsheet sort
Plus: How DOOM II and Nokia's N-GAGE each derailed Christmas
On-call Welcome again to On-Call, the column on which we conduct a Friday forage through the inbox full of readers' stories of jobs gone wrong.
Today, we wrap our special week of Festive On-Call and so our topic is ... drumroll please ... Christmas.
Let's start with “Mike” who told us that “Back in the Christmas of 1995 I was working for a call center who were handling the technical support calls for the old AST range of home PCs at the height of their push into the UK domestic marketplace.”
Mike says he got lots of calls about AST's mistaken use of US-style power cords in machines shipped to the UK. But aside from that things were pretty good.
At least until “a very unusual issue that came to light when the popular PC videos games of the day like Doom II started using the DOS 4GW library. Somehow they managed to over-write or corrupt the system BIOS. The game itself would run fine, but the next time you rebooted all the BIOS settings would revert to defaults.”
AST's default BIOS settings were OK, except for the PATA drive controller which did not automatically detect and mount drives.
“So the typical scenario was that the parents would buy and setup a brand new AST PC in perfect working order, then the kids would be let on to play their newfangled video games,” Mike explained to The Register. “The kids were, of course, told to turn if off when they had finished and in the morning when the parents turned it back on they would be greeted with 'Operating system not found' instead of a working Windows 95 PC."
It was Game Over for the PC until it was fixed, which took Mike about 30 seconds once customers made it through the 30-minute phone queue to reach him. And once he'd done his work, the data was restored.
But Mike can't help but think that “A lot of (mostly) innocent kids were banned from playing a lot of video games that Christmas.”
All thanks to the grinches at AST.
Nokia's notorious N-GAGE also killed Christmas
“Darren” wrote to tell us that “Back in the 90s I worked for a large US wireless telecom doing back office sales support for large corporate clients.”
The job had its moments: Darren told us that one of his clients was a forestry company and the telecoms manager “griped bitterly to me about the market shift from sturdier analog phones like the Motorola flips to the consumer-focused digital candy bar phones from Ericsson and Nokia.”
"Those phones can't handle 300lbs of logger's ass when they crawl under their rig to fix something!” Darren was told.
But we're using Darren's tale for a seasonal purpose, because one day in the office he heard a colleague from marketing excitedly saying the carrier had secured 50,000 of Nokia's N-GAGE is-it-a-phone-is-it-a-handheld-console devices, just in time for Christmas.
As The Register reported back in 2009, the N-GAGE cratered.
“I know the The Register (and others) have pondered in the past how many of those face tacos were actually sold/activated,” Darren wrote. “At this major wireless company, less than two dozen unique ESNs were ever activated.”
“And that included sales demos (but perhaps not test units).”
The spreadsheet that almost Grinched Christmas
Lastly, a story from “Gary” who told us that “A few years ago i was leading a small team of developers for a migration between two big ERP systems for an aircraft maintenance company.”
Gary and his team had the job of converting data from one system to the other, then uploading it.
“For reasons I fail to understand, the client manager was okay for a go-live date of December 24th,” Gary told us. As more than a few of the team wanted Christmas Eve off, “This was really not appreciated by the team, so we negotiated a few things as compensation.”
Those conditions included holiday pay, a job-done-down-tools-and-leave clause, with provision for no extra meetings or discussions. The team also wanted a full week of preparation for the job, with a working test system that could be restored to pristine condition every day.
Gary and his mates were granted their wishes, practiced the job until they could do it in two hours and showed up on Christmas Eve confident of a fast and lucrative getaway.
Come the day and the data was delivered, half an hour late.
And then the programs the team had developed to do the job decided not to work. Cue frantic examination of what was going on, followed by discovery of “function calls that had been removed, interfaces that ha changed, code that had been modified.”
Curiously, the code appeared to have been written by Gary and his team, but bore little resemblance to the tools they'd tested for the previous week.
Fortunately, Gary and his crew figured out what was wrong. The system they'd built relied on a spreadsheet to list modifications to the data. Someone had sorted that spreadsheet and in doing so mixed up the order in which files would be altered and migrated.
To this day, Gary can't imagine why someone would have altered that spreadsheet.
There's a happy ending here because Christmas was saved: after a brief meeting it was decided the migration could wait until January, Gary and his crew got to go home at lunchtime as planned and still got holiday pay for their half day!
“It was a happy, if stressful, Christmas,” he wrote.
Which looks as good a place to wrap on On-Call for 2016.
The column will be back in mid-January 2017, when the bulging On-Call inbox will again be delved into for weekly tales from The Register's wonderfully generous and witty readership. May the festive season bring you and yours happiness, good health, and no calls requiring you to do nasty jobs in nasty places at nasty times of the day or night in nasty weather. But if you do get called out over the next few days and want to tell us what happened, feel free to click here to share your story and put yourself in with a chance of appearing in a future edition of On-Call. ®
Sponsored: Becoming a Pragmatic Security Leader