Telstra issues report on Warrnambool exchange fire
Disaster recovery 'designed in real time'
Telstra has determined that the catastrophic fire in its Warrnambool Exchange, which in late 2012 took 100,000 Victorians offline, was an accident.
The carrier has issued a report on the incident, the summary of which says an unspecified electrical fault was the cause of the fire, either in the roof of the exchange's control room or inside the control room. While the fire control systems operated correctly, the fire still spread causing “considerable damage” to the exchange building and equipment.
Because of Warrnambool's status as a major local hub, 135 exchange service areas covering 26,000 square kilometres were impacted, with 85 schools, 20 hospitals, 27 police stations, 92 fire stations and 14 State Emergency Service locations also sent into a communications wilderness.
Recovery took 20 days to complete, during which local businesses lost more than $400,000 each day as a result of the crisis, according to the Victorian government. Telstra says that replacing the entire exchange would normally have taken months. Even now, full restoration to normal operations is expected to take as much as a further six months.
It appears Telstra is taking the event seriously: it has commenced a program of thermal imaging of the roof spaces of other exchanges “of similar significance” to detect any overheating cables or equipment; it is reviewing some key engineering standards and conducting building inspections.
All Ethernet aggregation points in regional Australia are to have specific disaster recovery plans, to aid recovery from similar events. As the report drily notes, “specific details” or the DRP for Warrnambool “were designed in real time during the service restoration phase”.
DC power system shutdown procedures will be reviewed, because the investigation revealed that the local fire service disconnected AC to the exchange, however “the DC power systems continued to supply the load which saw the smoke and soot drawn into sensitive equipment”.
While lightning wasn't identified as the cause of the fire, all similar exchanges will have their lightning control systems inspected, and cabling standards (such as in-building fibre diversity) will also be reviewed.
The exchange's maintenance control room before the fire ...
... and after. Images: Telstra
The full executive summary of the report, here, is probably worth a read for anyone responsible for disaster recovery. ®