Canadian ISP Telus launches novel solution to deal with excess email: Crash your servers and wipe it all

Dealing with email is possibly the most tedious daily exercise that the modern digital world has forced on us. But 13 million customers of Canadian ISP Telus have discovered that not having that problem is more of a burden.

Starting Thursday morning, the connection biz's entire email system went down, with not a single email in or out. Such outages happen occasionally but rarely last more than an hour. In Telus' case, however, customers really started getting annoyed when the issue still wasn't resolved that evening – eight hours later.

The ISP's support team tweeted a sad looking cheetah cub with the message "we're sorry" but that did little to dampen anger as the issue stretched in 24 hours. The next morning: still nothing.

Swamped with angry users, the internet provider explained that the outage had occurred as a result of some borked Dell-EMC-built storage equipment, and that in the course of repairing and replacing it, they accidentally took telus.net offline.

But as hours have bled into days, it has become clear that someone is responsible for some catastrophically bad system design. Despite getting the system back up and running for "more than 90 per cent of customers" on Friday, Telus was directing them to access their email by logging into their accounts through a browser.

For those able to login to webmail, however, they found an entirely empty inbox. "It's important to note that the webmail inbox will be empty except for new emails sent or received while we work to restore data from the old servers," Telus explained, strongly implying that it had lost all of its millions of customers' data and is having to dig into backups to rebuild it.

Not exactly the kind of thing you expect a huge ISP to need to do – as Telus itself has recognized: "As a technology company with a rich history in providing excellent customer service, we feel sick about this disruption and are doing everything we can to restore service as quickly as possible. Our team has been working nonstop since early Thursday morning and has all hands on deck to make this right."

What the Dell?

What the hell happened? Well according to a different explanation on the Telus' website: "This issue occurred during an overnight update to our servers in the early hours of Thursday, August 15, in partnership with our vendor Dell EMC, when a flawed repair procedure took the Telus.net email system offline."

So, basically, Telus tried repairing some servers with the help of Dell EMC, and lost all the data on it, taking everything down.

But the time Friday evening came around, nearly 48 hours after the initial cock-up, Telus' chief customer officer Tony Geherean – looking distinctly worn-out and non-plussed – was shoved in front of some office plants and told to apologize to camera.

"We've let you down and we are sorry," he began the e-mea-culpa. "I want to personally apologize for the extended extensive email outage that began on Thursday and prevented many of you from accessing, sending or receiving emails from your Telus.net account.

"We know how vital email is to conduct important business, to communicate with family and friends, and to manage schedules and finances. This disruption is unacceptable and we are doing everything we can to resolve the mater as soon as possible."

But he was also forced to acknowledge that 10 per cent of customers are still in the dark. "The issues with the remaining servers are complicated and is taking far longer to resolve this problem than we would like."

It really is. Fast forward to Monday morning – four days later – and this is the latest update. "Starting immediately, customers still experiencing issues can access their inbox by visiting webmail.telus.net from an Internet browser. Please note, the webmail inbox will be empty except for new emails you send or receive while we work to restore data and smoothly integrate the old and new mailboxes together."

Progress?

In other words, no clear progress has been made over the entire weekend. And Telus is now talking about "old and new mailboxes" suggesting a complete rebuild.

"Work is now underway to smoothly integrate the old and new mailboxes. Once this work is complete and the data is recovered, old messages will be restored to the mailbox and full access will be available."

"Our entire leadership team is committed to learning from this experience to augment the strength of our globally-leading networks and ensure our customers never find themselves in this situation again," Telus explained as part of its update.

In the meantime, we can only suggest that Telus customer shift to a third-party email provider that is far more stable. Like Google's Gmail… wait, what's that? ®