Twice-crashed HPE SANs at Oz Tax Office built for speed, not strength, and turned off error reporting
HPE settles after findings of badly fitted cables, bugs on disks and backup tools on SAN
Oz taxation commissioner Chris Jordan has revealed that the Australian Taxation Office (ATO) has reached a commercial settlement with HPE over the two outages to its online services caused by 3PAR storage arrays.
In remarks made to a Senate Committee today, Jordan said: "The turnkey service of data storage as per the 3PAR SAN provided by HPE failed us," and that the outages were attributable to three problems:
- The fibre optic cables feeding the SAN were not optimally fitted
- Disk drives on the SAN had software bugs that made the stored data on the drives inaccessible or unable to be read
- Some monitoring features were not activated, including a "back-to-base" tool to report operating errors
Jordan added: "The SAN design and configuration meant we had an overemphasis on performance features rather than stability or resilience – a relatively small disk drive failure had a large impact – only 12 of some 800 disk drives failed, but they impacted most ATO systems."
After the SAN failed, it didn't help that "some of the recovery tools required were stored on the same SAN that failed".
It appears that HPE has admitted fault to the tax office, as Jordan added: "We have reached a commercial settlement with HPE, the detailed terms of which are subject to contractual confidentiality. The settlement recoups key costs incurred by the ATO, and provides additional and higher grade IT equipment giving the ATO a world-class storage network."
More details will emerge next week, when Jordan said the ATO will release HPE's "interim root-cause analysis", the report into the incidents written by consultancy PwC and the ATO's own internal review.
The tax office's SANs first went down in December 2016, causing several days of disruption to online services. The SANs failed again in February. Jordan has previously said that hardware "dislodged" before the February failure.
The Register suggests readers do not stock up on popcorn ahead of next week's release of documents investigating the failure, as Jordan has previously warned they are likely to be heavily redacted. ®