Let's Encrypt? Let's revoke 3 million HTTPS certificates on Wednesday, more like: Check code loop blunder strikes

Tons of TLS certs need to be tossed immediately after Go snafu

By Thomas Claburn in San Francisco


On Wednesday, March 4, Let's Encrypt – the free, automated digital certificate authority – will briefly become Let's Revoke, to undo the issuance of more than three million flawed HTTPS certs.

In a post to the service's online forum on Saturday, Jacob Hoffman-Andrews, senior staff technologist at the EFF, said a bug had been found in the code for Boulder, Let's Encrypt's automated certificate management environment.

When someone asks Let's Encrypt for HTTPS certificates for their domain names, Boulder checks Certificate Authority Authorization (CAA) records to ensure the requests are all above board. The bug, introduced on July 25, 2019, was an error in the way the tool's Go code iterated over the domain names during this approval phase.

Free HTTPS certs for all – Let's Encrypt opens doors to world+dog


"The proximate cause of the bug was a common mistake in Go: taking a reference to a loop iterator variable," explained Hoffman-Andrews in the bug report.

So when Boulder iterated over, for example, a group of 10 domains names that required CAA rechecking, it would check one domain name 10 times instead of checking each of the 10 domains once.

"What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt," Hoffman-Andrews continued.

A code fix was deployed about two hours after the programming blunder was discovered, though that still leaves 3,048,289 digital certificates out of about 116 million that need to be revoked. About one million of the flawed set of certs are duplicates.

Affected certificate owners, who have supposedly been notified by email, have until 0000 UTC March 4 to renew and replace their certs. The process to do so for those using the Certbot command-line tool is simple in theory:

certbot renew --force-renewal

But reports of difficulties in the Let's Encrypt forum suggest not everyone will enjoy a trouble-free update process.

Come Wednesday, Let's Encrypt, which is supported by the Internet Security Research Group (ISRG), will revoke those certs that haven't been repaired, causing visitors at affected websites to see security warnings until the problem gets remedied.

For those who may have missed or deleted the notification email, Let's Encrypt has posted a list of affected serial numbers that can be downloaded. Concerned individuals can look up their account identifier(s) for associated certificate numbers. There's also a webpage for checking whether a site relies on an affected cert. ®

Sign up to our NewsletterGet IT in your inbox daily


Keep Reading

Microsoft brings K8s Security Center out of preview, replaces CoreOS Container Linux with Flatcar

Azure security dashboard now covers Kubernetes service - at a price

Google's OpenSK lets you BYOSK – burn your own security key

Now there's no excuse

SecureX gon give it to ya: Cisco muscles into the integrated security game

Push to get punters inhaling one cloudy product

US telcos tossed yet another extension to keep going with Huawei kit despite America's 'security threat' concerns

It's clearly not a pressing issue – this is the fourth time now

IoT security? We've heard of it, says waving new regs

Department of Fun straps on a holster, strides into the wild west of online gadget users

Staffer emails compromised and customer details exposed in T-Mobile US's third security whoopsie in as many years

And there it is – exactly what telco was fretting over in FY'19 results

Avast pulls plug on insecure JavaScript engine in its security software suite

Code interpreter ran with admin-level access, not sand-boxed, potentially open to remote-code execution

US Homeland Security mistakenly seizes British ad agency's website in prostitution probe gone wrong

They got it back – after reneging any claim against Uncle Sam for damages

Tech Resources

Why you need managed detection and response

How do you go about implementing MDR securely and manageably? Dave Martin from Open Systems has promised to tell us. He’s talking to the Reg’s Tim Phillips, and he will explain why your organization needs MDR, how to convince the business that it needs it too, and how to implement it.

KEMET Customer Story

KEMET chooses Open Systems to optimize performance of cloud apps, secure cloud connectivity and reduce costs by 50%

Faster Response with CrowdStrike and MITRE ATT&CK

Today’s threat landscape has created new challenges for security analysts and incident responders.

A Definitive Guide to Understanding and Meeting the CIS Critical Security Controls

The CIS Critical Security Controls are the industry standard for good security. Are you up to par?