A sysadmin in telco hell
1-800 oh my £&$%&
At the best of times, telecommunications is a complicated field to navigate. Putting aside the technical difficulties of creating and maintaining a modern telecommunications system, customer support and regulatory compliance can be challenging burdens for organisations of any size to cope with. Even with the best and brightest minds on your staff, errors occurring at layers eight through ten of the OSI model can and do occur.
The adventure begins with me in mop-up mode. The company closed two locations, so I am consolidating the IT and overseeing the reorganisation of our telecommunications infrastructure. The net result of the reorganisation is a move away from our archaic phone systems and Bell Canada's provisioned service to a hosted voice provider, Planet Telecom.
Somewhere around lunchtime on Thursday I am informed that, apparently, none of the company 1-800 numbers are working. With the recent store closures in mind, customers are emailing us – panicked – trying to find out if we were still in business. Not good.
I pinged Planet Telecom to see if something had gone wrong with the porting. They are in the dark; their systems are verifiably configured to start making use of the 1-800 numbers just as soon as they start hitting the system. Being the good folk they are, they dropped everything and immediately started phoning everybody they had numbers for, trying to find out what has gone wrong.
For my part, I spent the next day and a half trying to get hold of a warm body at Bell willing and empowered to help me solve this problem. The end of business on Friday came and went. Eventually, the general manager of the new telco met with limited success. He had been told by Bell that the new numbers now somehow belonged to AT&T, and we would have to contact AT&T for assistance.
Both Planet Telecom and I were completely flabbergasted. At no point did we understand AT&T to be involved in any way with the provisioning of these services. AT&T told us that we were not its customer and thus it could not help us.
It is at this point that I simply gave up trying to do things through the proper channels. I set up temporary 1-800 numbers so that I had something to put on the website. I then filed a complaint with the CRTC and carbon-copied the weekend press officer at Bell. 1
Emails launched, I packed it in for the weekend and went home. By Monday morning, the issue had been resolved.
The post-mortem took a while, but when the dust had settled the error was only partly Bell's in nature. Bell had received and acted upon the porting request, transferring the 1-800 numbers as it had been requested to do so. But here is where it gets complicated.
Planet Telecom had properly configured its systems to terminate the new numbers. However, it purchases service from an intermediate carrier (ThinkTel), which in turn apparently purchases its 1-800 provisioning in bulk from AT&T. AT&T's 1-800 provisioning in Canada is provided by ... Bell. (Yay, telecom!)
It was ThinkTel that had screwed up. It didn't provision its systems to accept the 1-800 numbers (from AT&T through Bell) and forward them to Planet Telecom 2. While the technical glitch that made the phones stop working had occurred with an intermediate provider, I retain a great deal of discomfort regarding Bell's handling of the situation.
As the end customer, I have no business relationship with ThinkTel, AT&T or anyone who might be involved in the provisioning chain. I have business relationships with Planet Telecom and with Bell. The Planet Telecom guys were available 24/7, and spent an entire weekend hauling the proverbial trying to get this fixed.
I was never able to reach an individual at Bell by phone willing and empowered to even attempt a resolution. Far more damning, it became apparent that Canada's largest phone carrier had somehow managed to lose the chain of custody on a routine phone number port. They couldn't tell me that a port request had been made, that it had been fulfilled, or who the number was now pointing at. As far as Bell's staffers were concerned, the number simply didn't belong to them, and thus it wasn't their problem.
It took the intervention of an individual fairly high up the food chain at Bell to focus efforts and finally figure out that the technical glitch occurred at the intermediary carrier (ThinkTel). Had he not been amenable to orthogonal problem resolution (and communication), who knows how long it would have taken to achieve resolution?
Some lessons were learned. Before porting phone numbers between carriers, take the time to determine the exact routing of those phone numbers. Find out which carrier "owns" the number, and how is it provisioned: from wholesaler through all intermediaries to the phone on your desk. Also take the time to hunt down exactly what the right phone number and combination of menu options are required to get to the appropriate tier of tech for the operation you are going to attempt.
It can take a lot of time to find this out – telcos are notoriously resistant to disclosing this information – but without this knowledge, your ability to troubleshoot these sorts of failures is effectively zero. Trusting that the proper channels established by your telco will provide a resolution can be a critical mistake.
Additionally, get temporary numbers set up before attempting a port, and make sure they are already on the web page. Just in case. ®
1 Even if the issue that my company was experiencing didn't see prompt resolution, I hoped to find out how this all went sideways so as to help some of my readers avoid a similar issue. Bell declined to comment.
2 Naturally, ThinkTel had been chased as a possible fault point by Planet Telecom from the start. However it wasn't until we were able to provide email traffic from Bell stating that the error absolutely must lie with ThinkTel that it was able to find and correct the error.