Original URL: https://www.theregister.com/2013/04/16/interview_dave_berstein_nbn/
Interview: Dave Berstein on the NBN, construction and VDSL
xDSL, fibre, construction and gigabits
Posted in Legal, 16th April 2013 01:03 GMT
Dave Berstein of DSL Prime and Fastnet News visited Australia during April for the CommsDay Summit. The Register interviewed him to get his take on the ongoing NBN debate.
The Register: Let's look at what you've seen of the two-headed broadband debate in Australia – how well do you think the shift from an FTTP to an FTTN model can be accomplished?
Dave Berstein: There's no problem on a technical level. It's all IP at this point – it doesn't matter what's at the end of the network. Everything talks to everything if they've designed it well, and they do have some competent engineers designing this stuff.
The political problems? The government problems? The operational problems? Australia has shown much of what can go wrong.
The Register: I presume that some of the things that can go wrong are independent of what your physical layer is. What can go wrong, if it's political or operational, isn't “because it's fibre” or “because it's copper” - it's because of decisions above that, management and so on. What are the potential problems that are common, whatever your technology?
Dave Berstein: Politicians promising what they can't deliver, managers of telcos promising to deploy faster than they can train people to do the deployment, numbers that were made up for political reasons that are so phoney they distort your operations, and newspapers looking for stories that will sensationalise even the most trivial nosense.
The Register: In the proposals that you've now heard … outlined by the Australian opposition. In the timetable, possibly the troubling aspect is the same thing that is now plaguing NBN Co – and that is that the political timetable is difficult from a construction point of view. In the case of NBN Co, it was the political promise to roll out a certain number of fibre areas at a particular rate that they have, so far, been unable to achieve. If I look at a 2014 start date for a VDSL-and-fibre combination for the Liberal Party proposal, and about 60,000 nodes by the 2016 election – that's 30,000 nodes a year. That is also a quite rapid rollout …
Dave Berstein: But it's not nearly the problem they were facing on fibre. For the same reason that doing this is much cheaper – there's less construction. You don't have to do all the local construction to reach every home. You still have to run one set of fibre, one box in the neighbourhood. Everybody knows how to hook up copper to the box now, as opposed to some ridiculous failure rate in fibre splicing that they came up with …
And so, because there's less construction, it's not as likely to have the same rate of problems. Now, The NBN has proven that they can create some of the most extreme problems of any network build I've ever seen … but the technology is much simpler, there's much less construction.
It takes 18 to 24 months to train somebody how to do something right in this business, that's [my] experience. Which is common in a lot of businesses. You don't learn it in a classroom or a training course, you learn it by getting out there and doing it.
So – you get utterly clobbered until you've got a fair number of technicians. They now are starting to get there, and by 18-24 months from now, you'll probably be in good shape.
The evidence is good that once this is rolling, it will roll very, very fast. British Telecom is doing 100,000 homes passed per week. So that if they do a smart job, and get smart operational people, who know how to work with the employees, it's not unrealistic to do a hell of a lot by 2016.
The Register: In the last week or so, America's Fibre-to-the-Home Council has released its survey of its members. Its claim is that out of 350 fibre-owning ISPs in North America, the fibre opex is, on an average, 20 percent lower than for copper, ranging between 9 percent lower and 30 percent lower, depending on who reporting it …
Dave Berstein: It's remarkable how lose those numbers are. When they first started going out with fibre, they said that since it's a permanent installation, you don't have to go out into the field, people like Verizon were saying they'd save 70 percent. So I was amazed to find the fibre guys being proud that they're going to save 20 percent – especially because this is all brand-new network, with today's technology, than something that's 40 years old!
The Register: So what did we get wrong in our predictions ten, fifteen years ago, in how we thought it was going to play out? Where did that other fifty percent go?
Dave Berstein: I don't know. It probably was hyperbole in the first place, and people who wanted to justify fibre, who thought it would be great for everybody … getting too optimistic. Realistically, savings are real, but they're not enough to justify the fibre costs by themselves.
The Register: So it turned out to be a simple matter of reality [being] less than forecast.
Dave Berstein: I can tell you that in the places I know, like Verizon, they didn't come close to the savings they thought they'd have. On the other hand, as the fibre council is saying, there are real savings.
The Register: By the way … from the Australian point of view, the citations – the names that are dropped on us are Verizon, Comcast … Chattanooga Electric and so on. It was jaw-dropping to see a survey saying “three hundred and fifty fibre owners”. Where's the rest of that market?
Dave Berstein: There's a historical accident, that makes the US different from almost everywhere else. In most of the world, the government built the phone network, one phone network per country. In the US, back around 1913, there was a political deal under anti-trust, that AT&T could be where it was, but that it would have to let other companies set up elsewhere … so there are literally three or four thousand small rural phone companies, ten fairly big ones, but literally thousands with ten or five thousand customers, local people that developed half a century ago when they finally brought phones to the rural areas.
And we've had a subsidy program for the rural areas for political reasons that was so rich that these guys were always making fortunes.
So they're still with us – lots of little telcos in the US that … the current regime is trying to cut back the subsidy, but they're struggling. The rich little rural telcos … have more money than some of the biggest companies on Earth to spend per user, and they brought fibre to hundreds of small towns across the US.
I'm working with one doing 22,000 lines of fibre homes, at a gigabit, called Vermont Telephone, with nice subsidies.
The Register: So part of Australians don't understand about America is that it's a Balkanized market of many, many local monopolies.
Dave Berstein: Actually, the US has a big advantage over Australia – we have local duopolies. Cable to over 96 percent of the country. That makes a total difference, because cable is offering better than the old DSL.
Which is why the folks in rural areas, who had to compete with cable, and who had lots of government subsidy, were among the first in the world to go to fibre. And that – the US has 18 million lines of fibre, or something like that. Sixteen million are Verizon. A million of them are 2,000 here, 5,000 there, 10,000 there – we have all these examples, they tend to be rural areas, small companies, and not very good examples to tell you what's going to happen if a big thing like Australia's NBN jumps in.
The Register: Their example, in other words – the rural US example – only provides any kind of information for a rural Australia rollout. Where disruption is likely to be less, and the cost of construction is likely to be less.
Dave Bernstein: Well, I'm actually working on a rural rollout, Vermont. We do have the advantage that most of it's poles, we don't have to do as much digging. It does help – it's not the only factor. But even with poles it's coming in at three or four thousand dollars per home, for the rural deployment.
But Verizon is coming at $600 per home in a mostly suburban deployment. When you do the numbers on this, try to cost things out, you have to break it down into cohorts.
And you just have to be aware of this, and work it when you do your numbers.
That's why people like Turnbull can't give you a good number. He's saying “I'm going to do my cost benefit analysis” and the Australian press said “Where is it? Where is it?”
You can do pretty good cost analysis. Benefit analysis is all off-the-wall.
A cost analysis you can do – but it takes four engineers, working for a month, with an awful lot of raw data that the Opposition doesn't have, to do an honest cost thing. So I can come up with a general [guide], but if you want honest numbers on this – it's not going to come from an economist working for two days.
It's engineers working for months.
The Register: Which also illustrates what is the great risk with a fibre rollout in Australia – that being that the original budget was not crafted by engineers working for months.
Dave Berstein: The original budget was crafted under the directions of a politician who said “these are the numbers we have to meet. You guys figure out how we're going to say we're going to do it. It doesn't have to be real.”
And obviously it wasn't.
On the other hand, the fact that for the first year of the actual building, they didn't come close to their own numbers and targets – that says something went wrong operationally.
The Register: In your experience of watching rollouts take place in America – is there a predictable point of no return on a project. Not the “point of no return, we're going to complete the project no matter what”, but the “point of no return of things going wrong.” In other words, at what point do the problems accumulate so much that it's irrecoverable.
Dave Berstein: Look at it from another point of view: how do you do it efficiently to ensure that there are not too many problems?
Experience is overwhelming that if you try to do too much in the first two years, you're likely to get screwed. It totally blew out the economics in the Burlington, Vermont municipal build – Utopia, a set of cities that got together, and they came in at four or five thousands dollars. It was a city / suburban build that essentially Verizon was doing for $US1,200.
They had amateurs in charge, businessmen and politicians, not network builders. They tried to do it quickly so they wouldn't have problems, and the city of Burlington Municipal Utility is bankrupt right now, and there's more a fear around that municipal build-out.
On the other hand, Verizon came in on time and on budget, because – probably because there's some very competent pros at Verizon who know networks, and probably because the network guys told the president of the company (I had a chance to talk to them) that “no, you do not want us going any faster”.
So they – for the first three years – built slowly, and expanded only as fast as they could train the crews, the contractors, build the operating system, train the people to answer the calls on the help desk, make sure everything was working.
In fact, Fabio – who thought the fibre was great and wanted it yesterday – told an audience … in the third year that he wanted to do four million lines. And the network people said “no – we can't do more than three million this year without trouble. Talk to us next year.”
You need that discipline to understand, fibre especially which is lots of construction, lots of new problems, you have to take the time, build the systems, train the people, don't push it – and then when you have the trained people, have the systems, move ahead.
I sincerely hope that Malcolm Turnbull is listening – I've had the chance to tell this to him as well. Go very cautious, until you prove that things are running smooth. Once things are running smooth, yeah, the technology is ready to go very fast. British Telecom's doing 100,000 homes a week.
The Register: If those planning principles had been employed – in other words, “do not excessively front-load your rollout” – then you have a set of principles that would have worked better for fibre?
Dave Berstein: And the teams. Okay, a thing as big as Australia's national broadband rollout, is going to require ten thousand people who know everything that's involved in coordinating the latest software release that comes from your router vendor, to make sure that you don't knock out a quarter of your network.
And it takes a while for people to acquire those skills, and you can't learn it in a classroom, and you can't hire a pile of trainers and run them through in six weeks and expect … a lot of this, you can only learn by doing. You have to plan it that way, or you're only inviting trouble.
Fibre is the technology, wherever it's going – there's nobody in Australia who's done a big fibre rollout before.
So it's guaranteed you have a lot of learning to do at the beginning. The VDSL is probably going to have a lot less trouble, even though it is also somewhat new technology, just because there's much less construction to it.
The VDSL is efficient because it's mostly using the old network, and only a small part of it. The node box and the fibre behind it is new … so there's less to go wrong. So it'll probably have fewer problems.
The Register: In the fibre rollout, in the business plan for the fibre rollout – we saw what is not surprising, a very rapid escalation in per-premises cost for every percent over 90. The first 90 percent did not vary much about the median price per home. The 91 varied a lot, 92 varied more, 93 a lot more.
In VDSL – here I'm looking at the characteristics of Australia …
Dave Berstein: Anybody who tries to give you a good answer to this, beyond eight-five percent, is bullshitting you. Because nobody has done this kind of node rollout beyond eighty, eighty-five percent.
So they're guessing … for the last fifteen percent they're guessing. Now, the guesses may be fairly well informed. The US Broadband Plan had a team with seven very good people trying to figure out where the costs are, and what was involved in the rural rollout.
What they found was that the cost was utterly and totally brutal – for the last forty-four hundredths of one percent!
Whether the point at which the costs escalate is at eighty-six percent, or ninety percent, ninety-three percent, or ninety-seven percent – expert opinion would differ, and nobody who hasn't spent a week with hard data of where those homes are located and what the distances are, is giving you any answer that isn't bullshit.
I would guess that somewhere between eighty-five and ninety-five, your cost escalates. I wouldn't tell you whether it's eighty-eight or ninety-four without access to a hell of a lot more data than anybody except Telstra or the NBN has access to.
And it would take me a lot of work to give you a good answer.
The Register: There is an inflection point …
Dave Berstein: And there's the fact that the NBN did what looked like some fairly careful work to say “you'll get killed at ninety-three percent”, which is why that's the cutoff. They may be wrong, it may be ninety or it may be ninety-five, and hopefully they put good people on to deciding that before they put money into building.
And some of getting this right is to go out and build some of those, and get some experience from the field, based on the density of the housing, and see what it actually takes with Australian labour and Australian systems.
The Register: In data-gathering terms, it is not a bad idea to start with a mix of territories, as in fact they did?
Dave Berstein: The best experts in the world on this can't give you a good answer until they get out there, and get the data. So, getting the data is why I'm fudging and not giving you direct answers to your questions! Instead of thinking that some expert from Sydney, New York, Berlin or Paris can come in from outside and give you the answer.
This needs real people, real engineers, real world, and lots of experience.
The Register: Final question: There is, at each inflection point – something unexpected happens. History at least suggests that anyone who tries to do a linear projection of bandwidth requirements ends up being proved wrong.
The shift from dial to 256, brought unexpected things – there was Skype ...
Dave Berstein: P2P video. Netflix, and so on.
I can bring you the opinion of some of the very best people in the world on this subject. Because I've had a chance to research some of this!
We don't know.
Dave Farber – called the grandfather of the Internet, because most of the people who are called fathers of the Internet were his students – said “it is utter nonsense to say that we don't need a gigabit, because the experience is that we always need more bandwidth and we'll find a way to use it.”
The guy who built one of the largest gigabit networks in the world is telling me he's only been able to figure out one real application … any time soon … that's likely to need more than 100 megabits.
A network engineer who's looking at the 100 megabits said he's got a big family doing a lot of stuff, but he doesn't ever touch 75 megabits.
So – we don't know if the extrapolation of the curve which says we always need more and more bandwidth is realistic going 20 years into the future, or the practical experience with disk drives is more relevant.
For 20 years in this business, whatever disk drive you had, you filled and needed a bigger one. Until about five years ago, when disk drives got so big that most people don't wind up running out of space on their Terabyte or 2 Terabyte drives, unless they do a heck of a lot of video.
Or have two-and-a-half Terabytes of jazz …
But those are relatively few, and we see that in the declining sales of hard disk drives.
There was a point where you had just about what you needed – so, smart people disagree on this one, and I'll wait 'till I see.