Brit hosting provider tsoHost takes needleful of 'unauthorized code' to the servers, suffers week of outages

UK hosting provider tsoHost is recovering from a week of major service disruption, after discovering "unauthorized code" was "injected into servers" in one of its data centres.

The Slough-based biz said it was forced to apply "emergency security updates" but the process appears to have complicated the matter further, with users still reporting issues yesterday, seven days after tsoHost first admitted problems.

Palindrome fans at tsoHost claim to have more than 250,000 customers – offering everything from domain names to dedicated servers. The Slough-based outfit was established in 2004 and merged with Vidahost to create Paragon Internet Group in 2011; Paragon was acquired by Host Europe Group in 2015, only to be swallowed by American hosting giant GoDaddy for $1.8bn in 2017.

Trouble at tsoHost started last Monday, 10 June, when some of its customers complained their servers were offline. Fourteen hours later, those who moaned to the company directly received a cryptic email, which The Reg has seen:

A major tsoHost customer told El Reg of last week's missive: "Now it is not rare to get the odd downtime but what is odd here is that they are being very elusive on cause, solution and ETA etc... Feels ominous."

On Tuesday last week, tsoHost revealed the apparent root cause – the security breach it called "unauthorized code."

It is not known whether any customer data was compromised, though clients were encouraged to "monitor their accounts for any suspicious activity".

"We promptly secured the server and launched an investigation to determine the extent of the issue," the company said in a dashboard update.

"We are primarily focused on quickly determining whether there may have been unauthorized access to any data contained on your hosting server. Investigations of this nature are complex, but we will provide additional information as soon as we have completed our investigation. In the meantime, we encourage customers to monitor their accounts for any suspicious activity."

It then tried to patch the hole, which appeared to affect its cPanel hosting servers, a budget range priced at up to £18.02 per month + VAT: "We have identified a vulnerability, which will require us to perform an urgent security upgrade in order to prevent further issues," the company said in a dashboard update that has since been deleted.

"Multiple shared servers will be affected by this change and this can cause temporary disruption to your service.

"We apologize for any caused inconvenience."

On Wednesday, and Thursday nights, the company continued with its ruthless upgrade campaign. But it doesn't look like site outages ended on Thursday – although some customers reported getting service back for a day or two. Customers have been venting their frustration online as recently as yesterday.

Your update on 12th June states it would be resolved by tomorrow night, it’s the 17th today?! #poorservice #tsohost

— ★ Deborah Moses (@TheSilverZebra) June 17, 2019

Users' websites are likely retrievable: "We protect your precious data through free daily site backups and we keep that data for 30 days. Just in case," states the blurb for the cPanel hosting range. It also states that all cPanel hosting servers are located in an, er, "superiorly secure" data centre.

Strangely enough, the status dashboard entry for the emergency patching extravaganza has disappeared from the internet, but here it is, saved for posterity.

We have contacted tsoHost for more information and will update this story if we hear back. ®