Redbus offers 'sincerest apologies' for loss of service
Incident report in full
Redbus has offered its "sincerest apologies" for the loss of service following yesterday's power failure at its Harbour Exchange facility in London yesterday.
The statement issued this afternoon reads:
Incident Report for occurrence on Tuesday 1st March 2005 at Redbus Interhouse Facility, Harbour Exchange Square, London
Time of Incident
On Tuesday 1st March 2005 we were informed of a fault on our systems at approximately 09:56hrs. The fault was rectified 5 minutes later at approximately 10.01Hrs, after which an initial inspection of the floors commenced.
Background to the incident
A routine on load building generator test was being carried out at the Redbus, Harbour Exchange facility.
Cause of the incident
During the initial stages of the generator testing a fault occurred within the switchgear panel serving the power to the 8th & 9th data floors.
On immediate visual investigation, the nature of the fault was derived from a measurement type "C.T. coil" becoming displaced, which caused a short circuit condition within the closed 2,500 amp switchgear panel supplying the UPS modules. This resulted in the feeds to the UPS modules being dropped as the relevant protection circuits took over.
At this time the event was further complicated by a building fire alarm being activated which by law meant that all personnel had to immediately vacate the building.
The UPS system acted as designed and switched to battery and remained in this operation for a number of minutes, until the battery autonomy time had been fully utilised and the UPS module closed down its inverter.
There was no static bypass available for the UPS supporting the critical load due to the un-determined severity of the fault condition, which subsequently resulted in the supply to the critical loads on the 8th & 9th data floors being lost as the affected part of the board had to be safely isolated in order for the repair works to be initiated safely.
Rectifying the fault
Power was restored to the effected racks via a manual bypass onto raw mains at approximately 10:01hrs.
Investigations were immediately carried out on the faulty switchgear panel, the fault was isolated and the specialists implemented immediate remedial actions. The redundant system was fully restored at approximately 19:15hrs on Tuesday 1st March 2005.
On Tuesday 1st March 2005, isolated areas of the Harbour Exchange Square data center suffered a loss of service. The event effected isolated areas, the remainder of the data center remained unaffected.
The loss of service was due to the failure of a specific piece of internal component hardware. This has now been fully rectified and refitted by the specialist manufacturer.
Once all customers have been transfered from the present electrical distribution system onto the new upgraded systems, Redbus will be able to de-commission the existing infrastructure. This will not only provide a greater piece of mind but also greater scope for expansion in the future.
Redbus offer their sincerest apologies for the loss of service this incident represents.
Sponsored: Global DDoS threat landscape report