Feeds

A sysadmin in telco hell

1-800 oh my £&$%&

Designing a Defense for Mobile Applications

At the best of times, telecommunications is a complicated field to navigate. Putting aside the technical difficulties of creating and maintaining a modern telecommunications system, customer support and regulatory compliance can be challenging burdens for organisations of any size to cope with. Even with the best and brightest minds on your staff, errors occurring at layers eight through ten of the OSI model can and do occur.

The adventure begins with me in mop-up mode. The company closed two locations, so I am consolidating the IT and overseeing the reorganisation of our telecommunications infrastructure. The net result of the reorganisation is a move away from our archaic phone systems and Bell Canada's provisioned service to a hosted voice provider, Planet Telecom.

Somewhere around lunchtime on Thursday I am informed that, apparently, none of the company 1-800 numbers are working. With the recent store closures in mind, customers are emailing us – panicked – trying to find out if we were still in business. Not good.

I pinged Planet Telecom to see if something had gone wrong with the porting. They are in the dark; their systems are verifiably configured to start making use of the 1-800 numbers just as soon as they start hitting the system. Being the good folk they are, they dropped everything and immediately started phoning everybody they had numbers for, trying to find out what has gone wrong.

For my part, I spent the next day and a half trying to get hold of a warm body at Bell willing and empowered to help me solve this problem. The end of business on Friday came and went. Eventually, the general manager of the new telco met with limited success. He had been told by Bell that the new numbers now somehow belonged to AT&T, and we would have to contact AT&T for assistance.

Both Planet Telecom and I were completely flabbergasted. At no point did we understand AT&T to be involved in any way with the provisioning of these services. AT&T told us that we were not its customer and thus it could not help us.

It is at this point that I simply gave up trying to do things through the proper channels. I set up temporary 1-800 numbers so that I had something to put on the website. I then filed a complaint with the CRTC and carbon-copied the weekend press officer at Bell. 1

Emails launched, I packed it in for the weekend and went home. By Monday morning, the issue had been resolved.

The post-mortem took a while, but when the dust had settled the error was only partly Bell's in nature. Bell had received and acted upon the porting request, transferring the 1-800 numbers as it had been requested to do so. But here is where it gets complicated.

Planet Telecom had properly configured its systems to terminate the new numbers. However, it purchases service from an intermediate carrier (ThinkTel), which in turn apparently purchases its 1-800 provisioning in bulk from AT&T. AT&T's 1-800 provisioning in Canada is provided by ... Bell. (Yay, telecom!)

It was ThinkTel that had screwed up. It didn't provision its systems to accept the 1-800 numbers (from AT&T through Bell) and forward them to Planet Telecom 2. While the technical glitch that made the phones stop working had occurred with an intermediate provider, I retain a great deal of discomfort regarding Bell's handling of the situation.

As the end customer, I have no business relationship with ThinkTel, AT&T or anyone who might be involved in the provisioning chain. I have business relationships with Planet Telecom and with Bell. The Planet Telecom guys were available 24/7, and spent an entire weekend hauling the proverbial trying to get this fixed.

I was never able to reach an individual at Bell by phone willing and empowered to even attempt a resolution. Far more damning, it became apparent that Canada's largest phone carrier had somehow managed to lose the chain of custody on a routine phone number port. They couldn't tell me that a port request had been made, that it had been fulfilled, or who the number was now pointing at. As far as Bell's staffers were concerned, the number simply didn't belong to them, and thus it wasn't their problem.

It took the intervention of an individual fairly high up the food chain at Bell to focus efforts and finally figure out that the technical glitch occurred at the intermediary carrier (ThinkTel). Had he not been amenable to orthogonal problem resolution (and communication), who knows how long it would have taken to achieve resolution?

Some lessons were learned. Before porting phone numbers between carriers, take the time to determine the exact routing of those phone numbers. Find out which carrier "owns" the number, and how is it provisioned: from wholesaler through all intermediaries to the phone on your desk. Also take the time to hunt down exactly what the right phone number and combination of menu options are required to get to the appropriate tier of tech for the operation you are going to attempt.

It can take a lot of time to find this out – telcos are notoriously resistant to disclosing this information – but without this knowledge, your ability to troubleshoot these sorts of failures is effectively zero. Trusting that the proper channels established by your telco will provide a resolution can be a critical mistake.

Additionally, get temporary numbers set up before attempting a port, and make sure they are already on the web page. Just in case. ®

1 Even if the issue that my company was experiencing didn't see prompt resolution, I hoped to find out how this all went sideways so as to help some of my readers avoid a similar issue. Bell declined to comment.

2 Naturally, ThinkTel had been chased as a possible fault point by Planet Telecom from the start. However it wasn't until we were able to provide email traffic from Bell stating that the error absolutely must lie with ThinkTel that it was able to find and correct the error.

The Power of One eBook: Top reasons to choose HP BladeSystem

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
Attack of the clones: Oracle's latest Red Hat Linux lookalike arrives
Oracle's Linux boss says Larry's Linux isn't just for Oracle apps anymore
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.