This article is more than 1 year old
Florence and the Machines: Data centers brace as hurricane smashes into US coast
TImes get wet and windy – but bit barns keep on ticking
Hurricane Florence has now landed on US soil, bringing 100 MPH winds, torrential rain, and claiming at least four lives. Many residents have fled, though some can’t – because they are keeping the area’s data centers up and running during the carnage.
The Carolinas, on America's east coast, are studded with data centers full of networking gear and server machines, thanks to cheap land, relatively low wages, government incentives, and other factors. Apple, Google, and Facebook all have facilities there, as do a host of other tech companies, and all have been preparing to keep services up and running during the storms.
“We have a disaster coordinator whose sole job is basically to keep an eye on the weather,” Robert Fiordaliso, director of critical infrastructure at data center operator CenturyLink, told The Register on Friday morning.
“We start raising awareness to upper management and tracking possible weather events. We have different stages of activation for disaster planning – days out, landfall and impact – and have been preparing for Florence for two weeks, when it first started coming out from Africa.”
Before Florence hit, the CenturyLink team, as one example, was already hard at work. All facilities in the likely strike zone were visited by crews and checked out to make sure backup generators were tested and fully functional, reserve fuel tanks are topped up, and any infrastructure issues are sorted out well ahead of time.
Get juiced up
The biggest concern is power. Transmission lines are usually one of the first casualties, and all sites have to have their own backup generators. However, keeping them fueled is a major issue, and the team has a dedicated fuel coordinator who set up a resupply mission.
“I currently have 50,000 gallons of fuel on site to refuel generators,” Fiordaliso said. “In addition we have several 10,000 gallon tankers, or motherships, just outside the predicted storm strike zone, and those are serviced by smaller trucks containing 2,500 or 5,000 gallons that can be pulled to sites by trucks.”
Stockpiles
Portable generators are also stockpiled for the electricians, heating, ventilation, and air conditioning (HVAC) specialists and network engineers, who may be needed to fix physical problems on the site. The biz has also set up arrangements with local specialists and vendors who can be on call throughout Florence’s visit to the US.
The other big danger, particularly as Florence is a slow-moving storm that’s hammering down rain, is flooding. Facilities that may be at risk have their own pumping equipment and sandbags, and some have physical barriers to keep unwanted water out.
As the storm hits all eyes will be on CenturyLink’s custom software that tracks outages in real time and can reroute traffic through still-running nodes and assign the nearest repair crew to the suffering station, although staff have to be kept safe.
“I managed a facility in St Croix last year, when Hurricane Maria hit,” Fiordaliso said. “We had a team hunker down in facility itself because it was built to survive a Cat 5 storm. But otherwise we stage them at a facility or in a hotel near the outer limits of the storm’s predicted trajectory.”
The team doesn’t disband until the storm has passed, and all facilities are up and running on mains power. Then, after some much-needed R&R, the team begins planning for the next big event, be it weeks, months, or years away. ®
Biting the hand that feeds IT
