A ghoulish tale of pigs, devs and docs revived from the dead
Dr Frankenstein had it all sewn up, you know
Something for the Weekend, Sir? "My pages have come alive!" accused one from my pod of guinea pigs, unfeasibly.
This (as it turned out) not-so-singular anecdote from my murky professional experience working on large-scale content management projects came to mind this week while reading The Reg. It was triggered by that recent Who, me? story about the little tech support boy who kept his finger in the server dyke.
For those of you with a dedicated TL;DR shortcut on your keyboard, the story describes an all-too-familiar nightmare scenario in which catastrophic data-loss is averted by scurrying IT bunnies who were, as it turned out, also partly complicit in conjuring up the problem in the first place. And all the while, masses of innocent and vulnerable users remained blissfully unaware of the virtual klaxons, action stations and baked bean wrestling taking place behind the server room's hermetically sealed doors.
So, what was all that about a pod (surely "pen"?) of pigs? Well, it was once upon a night that I was on customer-facing support duty at a national morning newspaper predominantly read by middle-aged women.
Usually when someone yells "My pages have come alive!" it will be while I'm teaching one of my interactive content design courses. I take it as a compliment. Unfortunately, on this occasion, I was the night's assigned tech support noddy on deck for a crew of miserable newspaper production slaves shipwrecked on an island of desks at the world's end – i.e. near the chilly windows that didn't shut properly.
They were unhappy because they'd won the dubious honour of being the first of several similarly unwilling teams on the title to use the new content management system in a live environment.
Things were going well, at least as expected. So far that evening, I had been repeatedly hurled into a bottomless pit of frivolous support demands and whiny complaints that were increasing in both frequency and unlikelihood as we headed towards the night's deadline. So far, so normal: these shifts will invariably begin with lots of vocalised expressions of impotent annoyance, which ramp up as time-pressure increases.
To be honest, I was quite pleased with how well it was panning out. My little bay of pigs had been snuffling around me right from the outset, snorting out unsolicited and improbable observations about the pearly new system every few seconds. This meant there'd be fewer questions late on in the night shift – by which time my patience, people skills and limited capacity for self-restraint while holding a plastic spoon become dangerously depleted...
...in which case, hey pig, piggy, pig, pig, pig, all of your fears could come true.
I had already solved and summarily dismissed such pressing problems such as "My mouse isn't working" (you are looking at the File menu and clicking but your mouse pointer is at the opposite corner of the screen), "I can't see my pages" (that's because you have quit the application) and "When I select everything and tap Backspace, everything vanishes" (yes it does that).
What could possibly top these? Well, having Dr Frankenstein in the corner claim that his pages were alive – ALIVE, I tell you! – was a bit much at 9pm when I wasn't even due to begin sipping from my support-shift hipflask for another half an hour.
I dragged myself over to his desk for the 100th time and asked him to demonstrate. What I witnessed was hardly Promethean. But it was curious.
Earlier in the shift, the good doctor had been complaining that his software had been bombing out and failing to reopen the pages he'd just been working on. On every occasion, I'd report the incident to the core dev team hidden away in a secret room elsewhere in the building, far removed from the ire of frustrated users. And on every occasion, the team would tell me it would be fixed "in a short while".
Now that Dr F had allowed "a while" to pass, he'd begun reopening the naughty pages, only to find that the pages didn't look the same as he'd left them.
My immediate thought was that the server had intercepted the crashed documents and was restoring earlier versions. But Frank handed me printouts he'd made of the pages earlier on and they bore only a passing resemblance to what I could now see on his display.
On his restored pages, text columns no longer lined up; pictures were the wrong way around; colours had changed; the dateline was upside down and back to front. It was as if the content had woken up and walked around the page by itself. It was aliiiiive!
Worse, my other poor guinea pigs were starting to suffer from the same thing. It was no longer a case of just a few files dying, reviving and spontaneously achieving self-awareness: now everyone at this pod of desks found themselves desperately fighting off hordes of the walking dead shuffling around their pages.
I called the dev team yet again and this time asked whereabouts they were situated, suggesting that I could bring over some printouts to help explain the problem. An evidently stressed young fellow blurted out the room number before the phone was very roughly snatched from the side of his head. "No need," another panicky voice announced, "we have printouts from the CMS already."
Thinking it rather unfair for a customer-facing support minion to be given the brush-off by fellow support minions sitting safely out of harm's way, I picked up my plastic spoon and headed off to find their room.
It turned out to be less of a room than an elongated cupboard, into which were squeezed eight shouting, sweating CMS developers whose eyes were fixed in 1,000-yard stares and whose brows would be wrinkled permanently after this night of hell.
A3 printouts were scattered across desks and the floor. One developer in a mad hurry tried to toss a used printout over his shoulder, and failed, because it fluttered back down to rest on the top of his head. But he was too busy to do anything about it, so the sheet remained there, held in place by the still-warm toner fusing with his hair gel.
And so I learnt the awful truth. The pages that were crashing my users' software were utterly and irrevocably irretrievable. The dev team had tried every trick to get them back and failed, and now that they had run out of time as the night's deadline loomed, switched to Plan F.
Plan F is the one in which you screengrab thumbnails of lost documents from the CMS browser window, print out the blocky pixellated grabs as large as you dare, then try to rebuild the originals yourself from scratch using pro-class design software with which you have merely a passing acquaintance in the forlorn hope that no one on the main floor finds out what's happening. As you might expect, the result is a load of pages that look a bit like the ones that were lost, but not the actual ones.
Without uttering a word, I stepped back and quietly closed the door. I tiptoed back to my users, told them the problem would be fixed "in a short while" and joined in the sly effort of rebuilding the pages.
We hit deadline – just – and the devs had the issue sorted by the following afternoon. Only then were they allowed to go to bed.
There is no moral to the story, no lesson to be learnt – unless "don't fuck up" is a lesson that you can put into practice by sheer will alone. But I find it comforting to have seen the wiring under the board for a fleeting moment: that behind every smooth operation is a cupboardful of IT bods frantically conducting a real-world and real-time disaster limitation exercise.
Any guinea pigs reading this: you have it bad for sure, but show respect for the live tech fixers because they have it worse. Don't bite the hand tossing you those pearls.
Sponsored: Becoming a Pragmatic Security Leader