Cloudflare hits the deck, websites sink from sight after the internet springs yet another BGP leak

Ghost in the machine conspires to ruin CDN biz's 10th birthday, it seems

Updated US network services provider Cloudflare has been celebrating its impending tenth birthday with a good, old-fashioned TITSUP*, er, knees-up.

Festivities began at around 1100 UTC this morning with website owners around the world noting that their sites had mysteriously become inaccessible for some users.

The issue, according to Cloudflare, is "a possible route leak impacting some Cloudflare IP ranges", which the company is feverishly working to fix. Its status page has, at least, remained more or less operational, even if hasn't really told worried users what has actually happened.

Cloudflare, while not a hosting provider, is the source of all manner of services needed by websites. It provides a content delivery network, an authoritative domain name system as well as load balancing, routing and DDoS protection and firewall services.

Thus when the company (or its network provider) has a wobble, those who depend on its services may also suffer a totter. The totter will not be at the server end, however, but rather on the route you are taking to get to the server. For example, El Reg – which is a customer of Cloudflare – is a bit up and down when accessed from some ISPs but not others.

Like at other sites, there is little our hardworking backend vultures can do about it until Cloudflare gets its act together. Fortunately the pubs are open, so there is that.

Cloudflare has said it is working with the "network involved" to resolve things, indicating something has gone awry at a lower level. Either some good old-fashioned finger trouble resulting in some network blocks being pointed the wrong way or maybe, just maybe, a bit of sabre-rattling hijack hijinks.

Don the tin-foil hat and fire up the speculatogun!

We're inclined to lean toward cock-up rather than conspiracy. However, the problem does highlight the fragility of some of the internet's infrastructure. It is, after all, held together mostly with duct tape, spit and trust.

We've contacted Cloudflare to get the technicals on the issue and will update this article when the company responds. ®

* Traffic Is Totally Screwed Up, Pal

Updated to add at 1344 UTC

Cloudflare has been in touch to let us know it was all caused by a classic BGP screw-up by someone else: "Earlier today, a widespread BGP routing leak affected a number of internet services and a portion of traffic to Cloudflare. All of Cloudflare's systems continued to run normally, but traffic wasn't getting to us for a portion of our domains. At this point, the network outage has been fixed and traffic levels are returning to normal.

"BGP acts as the backbone of the internet, routing traffic through internet transit providers and then to services like Cloudflare. There are more than 700k routes across the internet. By nature, route leaks are localized and can be caused by error or through malicious intent.

"We've written extensively about BGP and how we've adopted RPKI to help further secure it."

Final update

We spoke to Cloudflare's CTO who revealed more details on today's disruption, including how US telco Verizon effectively poured gasoline over the situation.

Disclosure: The Register is a Cloudflare customer.




More from The Register

Biting the hand that feeds IT © 1998–2019