Let's Encrypt? Let's revoke 3 million HTTPS certificates on Wednesday, more like: Check code loop blunder strikes
Tons of TLS certs need to be tossed immediately after Go snafu
On Wednesday, March 4, Let's Encrypt – the free, automated digital certificate authority – will briefly become Let's Revoke, to undo the issuance of more than three million flawed HTTPS certs.
In a post to the service's online forum on Saturday, Jacob Hoffman-Andrews, senior staff technologist at the EFF, said a bug had been found in the code for Boulder, Let's Encrypt's automated certificate management environment.
When someone asks Let's Encrypt for HTTPS certificates for their domain names, Boulder checks Certificate Authority Authorization (CAA) records to ensure the requests are all above board. The bug, introduced on July 25, 2019, was an error in the way the tool's Go code iterated over the domain names during this approval phase.
"The proximate cause of the bug was a common mistake in Go: taking a reference to a loop iterator variable," explained Hoffman-Andrews in the bug report.
So when Boulder iterated over, for example, a group of 10 domains names that required CAA rechecking, it would check one domain name 10 times instead of checking each of the 10 domains once.
"What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt," Hoffman-Andrews continued.
A code fix was deployed about two hours after the programming blunder was discovered, though that still leaves 3,048,289 digital certificates out of about 116 million that need to be revoked. About one million of the flawed set of certs are duplicates.
Affected certificate owners, who have supposedly been notified by email, have until 0000 UTC March 4 to renew and replace their certs. The process to do so for those using the Certbot command-line tool is simple in theory:
certbot renew --force-renewal
But reports of difficulties in the Let's Encrypt forum suggest not everyone will enjoy a trouble-free update process.
Come Wednesday, Let's Encrypt, which is supported by the Internet Security Research Group (ISRG), will revoke those certs that haven't been repaired, causing visitors at affected websites to see security warnings until the problem gets remedied.
For those who may have missed or deleted the notification email, Let's Encrypt has posted a list of affected serial numbers that can be downloaded. Concerned individuals can look up their account identifier(s) for associated certificate numbers. There's also a webpage for checking whether a site relies on an affected cert. ®