Comodo database glitch causes billing problems
Enter the snag-dragon
Updated While the rest of the world had its eyes firmly on the WannaCrypt outbreak, digital certificate firm Comodo suffered an unrelated but protracted database problem that affected its billing systems.
The Register learned of the issue from reader Ian Barber who came across the problem in the process of getting a new SSL certificate from Comodo activated last Friday. "It appears that Comodo having some issues. The scary bit is where they say they have restored to a database nine days old," he told El Reg in reaction to an emailed alert on the issue he received from Comodo – an extract of the missive below:
We regret to say that, due to a database system error, Comodo’s CA license database is having to be being restored. The initial restore has already taken place and all orders placed before 03-May-2017 12:19:52 UTC are being correctly managed.
Some orders placed after 03-May-2017 12:19:52 UTC may not be obtainable. We are currently working on resolving this. We do understand your situation. We sincerely apologize for the inconvenience caused.
The day before (on Thursday) a digital certificate reseller had noted an "unscheduled outage" of of Comodo's CA (certificate authority) billing service because of database problems.
In response to queries from El Reg, Robin Alden, Comodo’s chief technical officer, offered a detailed explanation of what had gone down. He said that although "Comodo has a generally good track record of minimizing unscheduled maintenance" it had issues with its billing systems that are yet to be fully resolving, adding that the operation of the certificates themselves was unaffected by the snafu.
Late last week, we identified an issue with a database that deals with Comodo’s orders and billing for the certificates and some related services.
Multiple errors in a 24-hour period had rendered the database corrupt.
It took a couple of iterations of restoring from backups to get the database working again.
The certificates themselves are present in multiple systems, so they are never lost.
Our ordering and billing system was unavailable for about 20 hours.
When the ordering and billing service was restored, there were some data gaps remaining due to the nature of the restoration. We are still backfilling that data, and some customers were unable to self-manage the lifecycle of some recently issued certificates. We regret the inconvenience of customers having to use our support services to manage lifecycles in some cases.
We anticipate that the backfill will be completed within the next 24 hours.
We apologize to our customers for any inconvenience this service interruption may have caused. Obviously, any service outage at all is too many.
El Reg had follow-up questions about how many Comodo customers were affected by this issue and what effect will they have seen. Comodo's PR acknowledged our query on these points but we had yet to hear back from the US-based computer security firm at the time of publication.
We'll update this story once we hear more. We passed Comodo's response back to Barber who had his own queries about how often the firm runs backups and how much data was lost because of its recent problems. ®
Updated to add
Robin Alden, Comodo's chief technical officer, has been in touch to say:
We have mirroring capability, which means things are backed up instantly. We also have daily backup on top of mirroring. The issue was that the bug affected the mirroring system as well. Hence, we had to go back to daily backups. These types of database bugs are not like computer crashes. They build up over time. So we had to go back to a time where the database was free of this bug.
The database that holds information on who paid what and when is not related to the ‘certificate database.’ No data was lost there.
There are many sources of data that are enabling us to rebuild the licensing database. It is just an annoyance and will take time, as things have to be carefully checked.
This issue only affects financial records in our licensing system, which could be considered at best as a ‘very low level delay’ to financial records.
There is no connection to WannaCry.