A sysadmin in telco hell

1-800 oh my £&$%&

Internet Security Threat Report 2014

At the best of times, telecommunications is a complicated field to navigate. Putting aside the technical difficulties of creating and maintaining a modern telecommunications system, customer support and regulatory compliance can be challenging burdens for organisations of any size to cope with. Even with the best and brightest minds on your staff, errors occurring at layers eight through ten of the OSI model can and do occur.

The adventure begins with me in mop-up mode. The company closed two locations, so I am consolidating the IT and overseeing the reorganisation of our telecommunications infrastructure. The net result of the reorganisation is a move away from our archaic phone systems and Bell Canada's provisioned service to a hosted voice provider, Planet Telecom.

Somewhere around lunchtime on Thursday I am informed that, apparently, none of the company 1-800 numbers are working. With the recent store closures in mind, customers are emailing us – panicked – trying to find out if we were still in business. Not good.

I pinged Planet Telecom to see if something had gone wrong with the porting. They are in the dark; their systems are verifiably configured to start making use of the 1-800 numbers just as soon as they start hitting the system. Being the good folk they are, they dropped everything and immediately started phoning everybody they had numbers for, trying to find out what has gone wrong.

For my part, I spent the next day and a half trying to get hold of a warm body at Bell willing and empowered to help me solve this problem. The end of business on Friday came and went. Eventually, the general manager of the new telco met with limited success. He had been told by Bell that the new numbers now somehow belonged to AT&T, and we would have to contact AT&T for assistance.

Both Planet Telecom and I were completely flabbergasted. At no point did we understand AT&T to be involved in any way with the provisioning of these services. AT&T told us that we were not its customer and thus it could not help us.

It is at this point that I simply gave up trying to do things through the proper channels. I set up temporary 1-800 numbers so that I had something to put on the website. I then filed a complaint with the CRTC and carbon-copied the weekend press officer at Bell. 1

Emails launched, I packed it in for the weekend and went home. By Monday morning, the issue had been resolved.

The post-mortem took a while, but when the dust had settled the error was only partly Bell's in nature. Bell had received and acted upon the porting request, transferring the 1-800 numbers as it had been requested to do so. But here is where it gets complicated.

Planet Telecom had properly configured its systems to terminate the new numbers. However, it purchases service from an intermediate carrier (ThinkTel), which in turn apparently purchases its 1-800 provisioning in bulk from AT&T. AT&T's 1-800 provisioning in Canada is provided by ... Bell. (Yay, telecom!)

It was ThinkTel that had screwed up. It didn't provision its systems to accept the 1-800 numbers (from AT&T through Bell) and forward them to Planet Telecom 2. While the technical glitch that made the phones stop working had occurred with an intermediate provider, I retain a great deal of discomfort regarding Bell's handling of the situation.

As the end customer, I have no business relationship with ThinkTel, AT&T or anyone who might be involved in the provisioning chain. I have business relationships with Planet Telecom and with Bell. The Planet Telecom guys were available 24/7, and spent an entire weekend hauling the proverbial trying to get this fixed.

I was never able to reach an individual at Bell by phone willing and empowered to even attempt a resolution. Far more damning, it became apparent that Canada's largest phone carrier had somehow managed to lose the chain of custody on a routine phone number port. They couldn't tell me that a port request had been made, that it had been fulfilled, or who the number was now pointing at. As far as Bell's staffers were concerned, the number simply didn't belong to them, and thus it wasn't their problem.

It took the intervention of an individual fairly high up the food chain at Bell to focus efforts and finally figure out that the technical glitch occurred at the intermediary carrier (ThinkTel). Had he not been amenable to orthogonal problem resolution (and communication), who knows how long it would have taken to achieve resolution?

Some lessons were learned. Before porting phone numbers between carriers, take the time to determine the exact routing of those phone numbers. Find out which carrier "owns" the number, and how is it provisioned: from wholesaler through all intermediaries to the phone on your desk. Also take the time to hunt down exactly what the right phone number and combination of menu options are required to get to the appropriate tier of tech for the operation you are going to attempt.

It can take a lot of time to find this out – telcos are notoriously resistant to disclosing this information – but without this knowledge, your ability to troubleshoot these sorts of failures is effectively zero. Trusting that the proper channels established by your telco will provide a resolution can be a critical mistake.

Additionally, get temporary numbers set up before attempting a port, and make sure they are already on the web page. Just in case. ®

1 Even if the issue that my company was experiencing didn't see prompt resolution, I hoped to find out how this all went sideways so as to help some of my readers avoid a similar issue. Bell declined to comment.

2 Naturally, ThinkTel had been chased as a possible fault point by Planet Telecom from the start. However it wasn't until we were able to provide email traffic from Bell stating that the error absolutely must lie with ThinkTel that it was able to find and correct the error.

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Fat fingered geo-block kept Aussies in the dark
You think the CLOUD's insecure? It's BETTER than UK.GOV's DATA CENTRES
We don't even know where some of them ARE – Maude
729 teraflops, 71,000-core Super cost just US$5,500 to build
Cloud doubters, this isn't going to be your best day
Want to STUFF Facebook with blatant ADVERTISING? Fine! But you must PAY
Pony up or push off, Zuck tells social marketeers
Oi, Europe! Tell US feds to GTFO of our servers, say Microsoft and pals
By writing a really angry letter about how it's harming our cloud business, ta
SAVE ME, NASA system builder, from my DEAD WORKSTATION
Anal-retentive hardware nerd in paws-on workstation crisis
Astro-boffins start opening universe simulation data
Got a supercomputer? Want to simulate a universe? Here you go
prev story


Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Managing SSL certificates with ease
The lack of operational efficiencies and compliance pitfalls associated with poor SSL certificate management, and how the right SSL certificate management tool can help.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.