Fat-fingered Level 3 techie reduces internet to level zero: Glitch knocks out connections

Celebrates CenturyLink gobble by blacking out

A man passed out

A fat-fingered configuration blunder at Level 3 temporarily sent a chunk of internet connectivity in America and the wider world to level zero on Monday.

The global backbone provider admitted it was at fault for rolling outages that have plagued big patches of US this morning, West Coast time, particularly for Comcast subscribers. Netizens in Europe and Australia have, like their American cousins, reported problems accessing websites, viewing Twitter images, and so on, due to Level 3 breaking their routing.

"On Monday, November 6, our network experienced a service disruption affecting some customers with IP-based services," a Level 3 spokesperson told The Register.

"The disruption was caused by a configuration error. We know how important these services are to our customers. Our technicians were able to restore service within approximately 90 minutes."

The Register understands a route leak, caused by Level 3, led to today's widespread disruption.

A heat map of busted connectivity, produced by outage monitoring service DownDetecter, showed just how extensive the problem was in the Land of the Free earlier today:

Level 3 map with November 6 outage

Red is dead ... America's Level 3 outage map as of mid-day US Pacific Time

Level 3 is one of the trunk roads of the internet that links big networks: if it goes down, it takes nearly everyone else with it. As such, folks also reported massive connection problems with Comcast's home and business internet services here in the USA. The cableco has blamed Level 3 for knocking out its connections. Vonage, a VoIP provider that relies on Comcast, also experienced downtime today, briefly cutting off phones including those in El Reg's San Francisco office.

Comcast November 6 outage map

Comcast outage map via DownDetector

Meanwhile, Microsoft's Azure service reported a "network Infrastructure" error was impacting connections to and from its North Central US region. Microsoft has so far declined to comment. We understand that the disruption is not caused by Level 3, however: Redmond may have broken its internal routing, according to chatter among network engineers seen by El Reg.

Azure November 6 outage

Houston, we have a problem ... Azure's notification (Click to enlarge)

Word of the Level 3 outage comes just days after CenturyLink officially confirmed it had completed the acquisition of the backbone when US watchdogs signed off on the $34bn stock and cash merger deal.

The timing of the outage certainly will please CenturyLink as it works to integrate its newest property into what it plans to be a global network of services for enterprise customers. ®


Biting the hand that feeds IT © 1998–2017