Feeds

Router crash downs CloudFlare services

A lesson in disclosure

Boost IT visibility and business value

During Sunday, US time, prominent Web services outfit CloudFlare sent an instruction to its routers in response to an attempted DoS, and instead took down its own network.

In a rare example of detailed disclosure, the company has posted an explanation of what happened here.

The network collapse occurred, the company explains, after it detected an attempted denial-of-service attack against a customer’s DNS servers using packets that were between 99,971 and 99,985 bytes long – an oddity, CloudFlare notes, because that’s so much larger than the Internet’s typical packet length (500 – 600 bytes according to the company) and larger than the 4,470 byte maximum packet it allows on its internal network.

So it wrote a JunOS rule (CloudFlare is a Juniper shop) to drop the packets, propagated the rule to its routers – and for reasons unknown, that rule crashed all the routers at which the instruction arrived.

“Flowspec accepted the rule and relayed it to our edge network. What should have happened is that no packet should have matched that rule because no packet was actually that large. What happened instead is that the routers encountered the rule and then proceeded to consume all their RAM until they crashed,” the blog post notes.

The crashes happened in such a way, CloudFlare says, that the routers didn’t reboot automatically, which meant that they couldn’t be accessed remotely; and worse, those routers that did wake back up copped the entire traffic load, couldn’t cope, and crashed again.

Accounts covered by SLAs will get credits, the company says, and it is investigating the problem with Juniper. ®

The essential guide to IT transformation

More from The Register

next story
Déjà vu: Virgin Media jacks up broadband prices
Screw copper phone lines, we're UNIQUE, bleats telco
NBN Co claims 96 mbps download speeds for FTTN trial
Umina trial also delivers 30 mbps uploads, but exact rig used not revealed
UK fuzz want PINCODES on ALL mobile phones
Met Police calls for mandatory passwords on all new mobes
Netflix swallows yet another bitter pill, inks peering deal with TWC
Net neutrality crusader once again pays up for priority access
New Sprint CEO says he will lower axe on staff – but prices come first
'Very disruptive' new rates to be revealed next week
EE: STILL Blighty's best mobe network, says 'Frappucino' Moore
Fresh round of network stats fisticuffs possibly on the cards here
US TV stations bowl sueball directly at FCC's spectrum mega-sale
Broadcasters upset about coverage and cost as they shift up and down the dials
ROAD TRIP! An FCC road trip – Leahy demands net neutrality debate across US
You crashed watchdog's site, now time to crash its ears
Google's so smart it's discovered SHARKS HAVE TEETH
Congratulations, world media, for rediscovering submarine cable armour
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
7 Elements of Radically Simple OS Migration
Avoid the typical headaches of OS migration during your next project by learning about 7 elements of radically simple OS migration.
BYOD's dark side: Data protection
An endpoint data protection solution that adds value to the user and the organization so it can protect itself from data loss as well as leverage corporate data.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?