IBM Australia didn't stress-test #censusfail router and blocked password resets
IT Crowd jokes aside, who okayed this mess?
If Vulture South wasn't running short-staffed yesterday, we'd have stayed with the Senate Committee hearings inquiring into Australia's Census outage on August 9, and caught this gem:
“If we had our time again we would have tested a hard power it off, power it on that router, that would have discovered earlier that we had that reboot and configuration loading problem” – IBM Australia managing director Kerry Purcell.
Before we go full ”IT Crowd” with the “try turning it off and turning it back on again” line, let's relate the events of the night).
The Census website experienced heavy traffic from the start, which really shouldn't have surprised anyone. To prevent against international attacks, the Australian Bureau of Statistics wanted IBM and its network providers (Vocus-owned Nextgen Networks and Telstra) to block traffic that didn't originate in Australia – but that didn't work.
The heavy traffic load crashed the router, and there was a backup – but in configuring the backup, IBM hadn't conducted a full power-down-reload test. Big Blue had merely simulated the effects of a router shutdown.
On the night, when the backup router was brought up, it came without configuration – and that's what a proper pre-flight stress test (yank the power cable without warning) would have revealed.
The other key moment in yesterday's hearing came when the Prime Minister's special advisor on cyber-security, Alastair MacGibbon, owned up to what everybody knew: the systems should have been able to cope with an attack so small you'd be hard put distinguishing it from a normal traffic load.
“They were indeed small attacks”, MacGibbon said. “There was a massive difference between the size of the attacks on the Bureau of Statistics' census website and the ones that are encountered routinely by corporations and governments”.
IBM and the ABS managed to put in place another howler of a misconfiguration: “Island Australia” (which only partially worked) was designed to block traffic from outside Australia, but IBM was hosting the Census password reset facility offshore.
Australian media are reporting that Prime Minister Malcolm Turnbull has MacGibbon's report, and is considering which “heads will roll” over the debacle. ®