23,000 HTTPS certs will be axed in next 24 hours after private keys leak

Trustico, DigiCert come to blows as browsers prepare to snub Symantec-brand SSL

By John Leyden


Customers of HTTPS certificate reseller Trustico are reeling after being told their website security certs – as many as 23,000 – will be rendered useless within the next 24 hours.

This is allegedly due to a security blunder in which the private keys for said certificates ended up in an email sent by Trustico. Those keys are supposed to be secret, and only held by the cert owners, and certainly not to be disclosed in messages. In the wrong hands, they can be used by malicious websites to masquerade as legit operations.

Unless the affected certificates are replaced in time, visitors to websites using Trustico-sold HTTPS certs will be turned away by their browsers, due to the digital certificates being revoked.

The whole situation is a mess, and possibly the result of a turf war. Here's what we've managed to ascertain.

What is Trustico?

Trustico, based in Croydon, UK, touted SSL/TLS certificates, which are used by websites to encrypt and secure their connections. It resold certs from the Symantec brand umbrella: Symantec, GeoTrust, Thawte, and RapidSSL. This umbrella is now owned and operated by DigiCert.

If you wanted to buy, say, a RapidSSL-issued certificate, you could do so via Trustico. The HTTPS cert ultimately leads back, along a chain of trust, to DigiCert, a root certificate authority trusted by web browsers and other software. In turn, a website presenting the Trustico-sold cert is trusted, its traffic secured using encryption, and the reassuring green padlock is displayed in visitors' browsers.

Why are the certificates being revoked?

According to DigiCert's chief product officer Jeremy Rowley earlier today, Trustico told DigiCert in early February that its resold certificates had been in some way "compromised," and that the certs needed to be mass revoked as a result.

DigiCert staff, we're told, asked Trustico for more information on this security mishap. The reseller replied it had a copy of the private keys, which is usually grounds for revocation, and thus insisted that DigiCert revoke the certificates.

When pressed for evidence, Trustico on Tuesday simply emailed DigiCert 23,000 certificates' private keys as proof it held this information, it is claimed. This forced DigiCert's hand: under the rulebook of standards set by the elders of the certificate security and browser worlds, the Trustico-sold certificates had to be revoked as a precaution within 24 hours. Specifically, the ones with their private keys in the email will be canceled.

"Trustico has not provided any information about how these certificates were compromised or how they acquired the private keys," explained Rowley.

"As is standard practice for a Certificate Authority, DigiCert never had possession of these private keys. Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident."

On Twitter, Rowley continued: "I'll likely be posting the private keys later once people have a fair chance to replace their certificates ... The allegation of compromise, keys compromised, and request for revocation all came from Trustico."

Before you raise an eyebrow too high, by posting the private keys, Rowley plans to disclose self-signed certificates, produced using the private keys, to prove the secret information was sent to DigiCert without revealing the actual information in public. Some have already popped online as proof DigiCert received the secret keys from Trustico.

Alarm bells

To warn netizens to the upcoming mass revocation, DigiCert's RapidSSL business sent out email alerts to Trustico customers urging them to get new HTTPS certificates or watch their sites go dark. Here's a copy of the memo, passed to El Reg:

Red alert ... Click to enlarge

DigiCert also put out a blog post, giving its side of the story:

Trustico requested revocation of their Symantec, GeoTrust, Thawte and RapidSSL certificates, claiming the certificates were compromised. When we asked for proof of the “compromise,” Trustico did not provide details on why they were requesting the immediate revocation. Trustico’s CEO indicated that Trustico held the private keys for those certificates, and then emailed us approximately 20,000 certificate private keys.

When he sent us those keys, his action gave us no choice but to act in accordance with the CA/Browser Forum Baseline Requirements, which mandate that we revoke a compromised certificate within 24 hours. As a CA, we had no choice but to follow the Baseline Requirements.

Following our standard revocation process, we gave notice via email to each certificate holder whose private keys had been exposed to us by Trustico, so they could have time to get a replacement certificate.

Now, over to Trustico.

Upset and denials

We asked the Brit biz for comment, and had yet to hear back at time of writing. However, posting on Mozilla's security policy newsgroup, Trustico product manager Zane Lucas was clearly upset that DigiCert sent out the above alert.

"We didn't authorise DigiCert to contact our customers and we didn't approve the content of their email," wrote Lucas.

"At no time had any private keys been compromised, nor had we ever informed to you that any private keys had been compromised. During our many discussions over the past week we put it to you that we believe Symantec to have operated our account in a manner whereby it had been compromised. Your usage of the word compromise has been twisted by you to your benefit and is absolutely defamatory."

To put this in context: Trustico was fed up with using Symantec certs, and on February 13, it formally abandoned the umbrella of brands – ahead of Google Chrome and Mozilla Firefox officially distrusting the certificates due to past security fumbles by Symantec. Trustico said it had complained privately to Symantec of long-running concerns over the security safeguards on Symantec-branded of certificates, hence Lucas' reference to its Symantec account.

Although Lucas stressed the private keys for Trustico's resold certificates were not compromised, it did, according to DigiCert, email a copy of 23,000 of them to the root authority seemingly to trigger their revocation. At that point, DigiCert considered the certificates at risk, and started the countdown clock to cancel them.

Trustico and DigiCert have clearly majorly fallen out, with the pair going their separate ways this month amid the behind-the-scenes drama. It even appears Trustico tried to stop DigiCert from using its online portal to send out today's emailed warning.

In future, Trustico will flog Comodo HTTPS certificates rather than peddle Symantec-branded certs. Cynics have suggested the Brit reseller ordered the revocation of its Symantec-umbrella certs so it could drive its customers onto Comodo certificates, and thus avoid the looming Google Chrome HTTPS certificate apocalypse without losing many, if any, punters. In effect, website owners have been caught up in a turf war between Trustico and DigiCert.

How did Trustico get the private keys to certificates it resold? We don't know for sure – but it did, and still does, offer an online private key generator for certificates. Just saying.

In an email sent to customers a few hours ago, and seen by The Register, Trustico said it will provide free certificates to replace the soon-to-be-nuked SSL/TLS certs:

Recently we wrote to you to let you know that we are no longer offering Symantec, GeoTrust, RapidSSL and Thawte branded SSL Certificates. Unfortunately, Google Chrome has decided to distrust these SSL Certificates. It's important to us that you SSL Certificate continues to function as normal, and not be compromised by the distrust of the Symantec brands. It is now required that you replace any existing distrusted SSL Certificate with one that is trusted by all web browsers.

Rest assured, there hasn't been any type of compromise of our systems. However, Symantec brands will cease to function correctly due to Google Chrome's decision to distrust them.

Recently DigiCert acquired the Symantec SSL Certificate division and subsequently an e-mail was sent by DigiCert to some of our SSL Certificate customers advising of the revocation of their distrusted SSL Certificate. We didn't authorise this e-mail to be sent and had specifically disabled it within the DigiCert system. We understand that the e-mail sent about your distrusted SSL Certificates may be confusing. It's important that you take the opportunity to replace your SSL Certificate as soon as possible.

We're providing free replacement of affected SSL Certificates. To enable a free replacement, you'll receive an e-mail report today if you have affected SSL Certificates. Your report will contain a unique coupon code for each affected SSL Certificate. When you replace your distrusted SSL Certificates using your unique coupon codes you'll receive extra validity free of charge. If you have any questions please feel free to reply to this e-mail.

Meanwhile, DigiCert said it, too, will offer free replacement certs to folks using Symantec-branded HTTPS certificates, which will be ignored by web browsers later this year. And, of course, don't forget you can grab free HTTPS certificates from Let's Encrypt that all major browsers trust.

Today has been marred with confusion. Trustico's customer support lines have been jammed with complaints and queries, following DigiCert's email alerts. Reg readers told us they felt left in the dark. Perhaps it'll all be clearer in a few hours, when the dust has settled – and the certs have been nuked. ®

Updated to add

Trustico kept the private keys to its customers' certificates in cold storage, and provided them to DigiCert to start the revocation process.

Sign up to our NewsletterGet IT in your inbox daily



Symantec shares up as private equity suitors sniff consumer tentacle

$16bn slapped on table by Permira and Advent – reports

Accenture pays for CSS injection from Symantec parent Broadcom: Yep, it bought its cybersecurity services arm

Price tag undisclosed but we're guessing it won't have made seller rich

Ye olde Blue Screen of Death is back – this time, a bad Symantec update is to blame

Updated The wrong kind of intrusion protection

Cloud nine to cloud nein: Google beefs up punters' data encryption to fend off cyber-thieves

Just don't forget where you put your keys and you'll be fine

Interpol: Strong encryption helps online predators. Build backdoors

Multinational cop agency reportedly set to issue statement

Euro ISP club: Sure, weaken encryption. It'll only undermine security for everyone, morons

UK, Oz and US pleas to Facebook given short shrift

Symantec share price nose dives after rumored Broadcom biz gobble taken off the menu

Looks like the ailing security shop priced itself out of an acquisition by chip giant

When the chips are down, buy a software biz: Broadcom snaffles Symantec for $10.7bn

Legacy security outfit to vanish into the 'rightsizing' grinder

Americans should have strong privacy-protecting encryption ...that the Feds and cops can break, say senators

I don't care if it's mathematically impossible, make it happen nerds!

Symantec boss Greg Clark exits biz amid dismal financials

Troubled security house keeps up trend of sudden resignations


Delivering Instant Experiences: Optimizing the Performance, Cost and Capacity of Data-Driven Applications

How can you accelerate data processing to keep up with accelerating business demands for an instant experience? Get the answer to this question and more in this webinar.

CEO Fraud Prevention Manual

CEO fraud has ruined the careers of many executives and loyal employees. Don’t be one of them.

Fine turn multi-cloud with containers and Intel Optane DC

Intel’s paper Making Multi-cloud Work discusses the seven considerations IT chiefs should address when optimising their multi-cloud environment and it comes with two companion reports

The Rise of Machine Learning (ML) in Cybersecurity

While many are guarding the front door with yesterday’s signature-based AV solutions, today’s malware walks out the back door with all their data.