Cloudbleed: Big web brands 'leaked crypto keys, personal secrets' thanks to Cloudflare bug

Updated Big-name websites potentially leaked people's private session tokens and personal information into strangers' browsers, due to a Cloudflare bug uncovered by Google researchers.

As we'll see, a single character – '>' rather than '=' – in Cloudflare's software source code sparked the security blunder.

Cloudflare helps companies spread their websites and online services across the internet. Due to a programming blunder, for several months Cloudflare's systems slipped random chunks of server memory into some webpages, under certain circumstances. That means if you visited a website powered by Cloudflare, there's a small chance you may have ended up getting chunks of someone else's web traffic bunged at the bottom of your browser page.

For example, Cloudflare hosts Uber, OK Cupid, and Fitbit, among thousands of others. It was discovered that visiting any site hosted by Cloudflare would sometimes cough up sensitive information from strangers' Uber, OK Cupid, and Fitbit sessions. Think of it as sitting down at a restaurant, supposedly at a clean table, and in addition to being handed a menu, you're also handed the contents of the previous diner's wallet or purse.

This leak was triggered when webpages had a particular combination of unbalanced HTML tags, which confused Cloudflare's proxy servers and caused them to spit out data belonging to other people – even if that data was protected by HTTPS. The webpages would also need to use a particular set of Cloudflare services as a catalyst for the spillage.

Normally, this injected information would have gone largely unnoticed, hidden away in the webpage source or at the bottom of a page, but some leaks were spotted by security researchers – and the escaped data made its way into Google and Bing caches and the hands of other bots trawling the web.

Timeline

The blunder was primarily discovered by Tavis Ormandy, the British bug hunter at Google's Project Zero security team, when he was working on a side project last week. He found large chunks of data including session tokens and API keys, cookies and passwords in cached pages crawled by the Google search engine. These secrets can be used to log into services as someone else.

"The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to clean up," he said today in an advisory explaining the issue.

"I've informed Cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."

Ormandy said that the Google team worked quickly to clear any private information and that Cloudflare assembled a team to deal with it. He provisionally identified the source of the leaks as Cloudflare's ScrapeShield application, which is designed to stop bots copying information from websites wholesale, but it turns out the problems ran deeper than that.

Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. https://t.co/wjwE4M3Pbk

— Tavis Ormandy (@taviso) February 23, 2017

On Thursday afternoon, Cloudflare published a highly detailed incident report into the issue: it happens that Cloudflare's Email Obfuscation, Server-Side Excludes and Automatic HTTPS Rewrites functions were the culprits.

The problem occurred when the company decided to develop a new HTML parser for its edge servers. The parser was written using Ragel, and turned into machine-generated C code. This code suffered from a buffer overflow vulnerability triggered by unbalanced HTML tags on pages. This is the broken pointer-checking source that is supposed to stop the program from overwriting its memory:

/* generated code. p = pointer, pe = end of buffer */
if ( ++p == pe )
    goto _test_eof;

What happens is that elsewhere p becomes greater than pe, thus it avoids the length check, allowing the buffer to overflow with extra information. This eventually leads to the above web session leaks. We're told this bug is not Ragel's fault; instead, it stems from the way Cloudflare used the state machine tool, apparently.

"The root cause of the bug was that reaching the end of a buffer was checked using the equality operator and a pointer was able to step past the end of the buffer," said Cloudflare's head of engineering John Graham-Cumming in the biz's incident report.

"Had the check been done using >= instead of == jumping over the buffer end would have been caught."

According to Graham-Cumming, for data to leak, the final buffer had to finish with a malformed script or img tag, be less than 4KB in length (otherwise Nginx would crash), and be running the three functions.

The new cf-html parser was added to Cloudflare's Automatic HTTP Rewrites function on September 22, 2016, to its Server-Side Excludes app on January 30 this year, and partially added to the biz's Email Obfuscation feature on February 13. It was only in the Email Obfuscation case that significant memory leakage appears to have happened, which tipped off Ormandy.

Cloudflare does have a kill switch for the more recent of its functions and shut down Email Obfuscation within 47 minutes of hearing from Ormandy. It did the same for Automatic HTTPS Rewrites a little over three hours later. Server-Side Excludes couldn't be killed, but the company says it developed a patch within three hours.

Logs on Cloudflare systems show that the period of greatest leakage occurred between February 13 and 18, and even then only 1 in every 3,300,000 HTTP requests through Cloudflare leaked data. We're told the proxy server bug affected 3,438 domains, and 150 Cloudflare customers. The biz said it held off disclosing the issue until it was sure that search engines had cleared their caches. Ormandy reckons those caches are still holding onto sensitive leaked info.

Ormandy also noted that the top award for Cloudflare's bug bounty program is a t-shirt. Maybe the web giant will reconsider that in the future. ®

Full disclosure: The Register is a Cloudflare customer. You can find other sites hosted by Cloudflare listed here. If you use one of the affected websites, now would be a good time to log out or otherwise invalidate your session tokens, get new API keys if necessary, and log back in.

Updated to add on March 1

Cloudflare CEO Matthew Prince has blogged some more about the impact of Cloudbleed, attempting to down play the seriousness of the blunder in terms of the amount of sensitive information leaked. Prince reckons only a sliver of data was emitted by the parser bug. Code auditing biz Veracode is now scrutinizing Cloudflare's source.