Customer Success Testimonial: Recovery is Everything

A rogue space in the date field caused almost 30,000 Scottish students to get their exam results a day early, as Excel versions clashed and human checking fell down.

The texts telling students their Higher results (the Scottish equivalent of A-Levels) were supposed to go out Thursday morning, having been preloaded onto the automated systems at text-specialist AQL, as a CSV export of an Excel spreadsheet. The problem was a space which somehow got appended to the dates, causing Excel to export it as a text field, in quotes, which got rejected by the automated system which then substituted the default setting: the current date.

The vast majority of AQL's clients, which number more than 30,000, manage their requirements through a direct interface with AQL's Web Services back end. From there, they can send messages and manage campaigns without human interaction or the possibility of incompatibilities.

But the annual sending of exam results, which AQL does as a freebie, doesn't really need a direct interface, so each year the company sends the Scottish Qualifications Authority a template spreadsheet to be populated and returned. This year that spreadsheet came back as an "xlsx" rather than an "xls", but (more importantly) it also came back with a space appended to every date field, causing Excel to assume it was text instead.

So Excel dropped quotation marks round the field when exporting it, which caused the import to disregard the field and replace it with a default date – today (that day) – so the messages were instantly sent out.

Fortunately the damage was minimal. Students got their results slightly early but as the clearing system wasn't operational they couldn't do much with the information other than celebrate/drown their sorrows 24 hours earlier than they might have done.

AQL tells us the import software has already been fixed, so the combination of events can't reoccur. They'll be checking all their processes to find out why the last-stage human read-through didn't pick up the error, and see where else their old systems might need updating.

It's easy to say it shouldn't have happened, but as our own Verity Stob explained last week the rusting of software applications is inevitable as assumptions made during development become invalid. We'd like to think our own applications would flag such an apparently obvious corruption of the imported data, but there's nothing like watching someone else get it wrong to focus the mind and make sure its not us the next time around. ®

Agentless Backup is Not a Myth