Scoop! The inside story of the news website that saved the BBC
How BBC News Online was created from scratch
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
Special report Fifteen years ago this month the BBC launched its News Online website. Developed internally with a skeleton team, the web service rapidly became the face of the BBC on the internet, and its biggest success story – winning four successive BAFTA awards.
Remarkably, it operated at a third of the cost of rival commercial online news operations – unheard of in public-sector IT projects. Devised before there were really any content management systems, the technical architecture became a template for all major news systems, and one that’s still in use today. The team endured some furious internal politicking and sabotage to survive.
Here for the first time is the inside story of the website that saved the BBC – with contributions from key figures including former BBC director general John Birt, now Lord Birt.
The BBC reports its award-winning BBC site
Why the internet?
When the information superhighway arrived on a wave of hype, there was good reason to be sceptical. Dialup computer bulletin board systems (BBSes) had been around for a decade albeit with limited adoption from enthusiasts. The French version of Prestel, Minitel, had only reached its wide audience of about 9 million homes because the French government had subsidised the terminals. The internet remained just as expensive, at UK dialup rates, as the marginal BBSes.
“A senior news editor told me the internet would be a flash in the pan – this was in 1996,” recalled one BBC correspondent. When it was clear it wouldn’t, it was argued it would confuse people.
Another Beeb journalist, Mike Smartt, who was an early enthusiast of the online service, said: “Putting web links on TV would merely bamboozle viewers, they argued. Announcing them on radio news would be worse and take up precious time."
Fortunately, one BBC figure fascinated by digital technology was best placed to ensure the online operation could take off: Lord Birt, the director general of the corporation, who was an Oxford engineering graduate. At a grammar school in Liverpool in the late 1950s, he had been given a project to explain the difference between analogue and digital computers to his science class.
“I didn’t find a use for that knowledge for twenty five years,” he now chuckles.
Lord Birt had been trumpeting the impact of new technology, such as fibre-optic networks, and how it could dramatically lower the barriers to entry: TV could become a read-write medium, he pointed out in a 1979 lecture.
Although the newborn internet was on the Beeb's radar from early on, no one knew exactly what to do with it. There was a desire to explore it, Lord Birt told us, but “this was quizzical, curious and inchoate – it was a very long time before that coalesced into anything you'd call insight”.
For Lord Birt, the 'net looked like “a way of sending rather dry-looking text around the world speedily and it did feel all very techie – this wispy blue print, no pictures, no graphic design, certainly no moving pictures. No real impact. It felt like the kind of medium you'd use to send academic papers to a foreign university”.
Lord Birt
“This is a medium in which you can transmit news – but in what form – and to what market – these were things we didn't know the answer,” he told The Register.
In the summer of 1996, the BBC had signed a deal with Fujitsu ICL. For £50m, the computer giant acquired the right to use BBC material and build its websites – including a news service. The sites would use Fujitsu ICL software, under shelter under the umbrella domain Beeb.com. The service would be funded by advertising.
Lord Birt explained the thinking:
The early but faulty insight was that the internet was an extension of the world of publishing. What part of the BBC published magazines? That was BBC Worldwide. And we couldn't see, at that point, any real prospect of utilising the technology. That plainly was a mistake.
But then the DG had second thoughts. Serious second thoughts.
"The story has many heroes and one is Jeremy Mayhew," said Lord Birt. Mayhew had been an advisor to the MP Peter Lilley, who was a member of the Tory Cabinet in the early 1990s, before joining the BBC Strategy Unit in 1993. He became the director of new media two years later.
“He was the one who did the deal with ICL which was very much again a piece of venturing on both sides,” added Lord Birt. “In life sometimes you make a decision instantly – I looked at Mayhew's [online news] prototype and” – the peer clicks his fingers – “I saw what the implications were. I saw we had made a big mistake and that we needed to do a U-turn – and it was unthinkable you could put that service like that out with advertisements.
“All of those thoughts occurred to me in half a second – and all credit to Jeremy for the work – all of those things became instantly apparent.”
Lord Birt decided he had to take the news website out of the Fujitsu ICL deal: the BBC must own its own internet news site.
“That was really bad news for ICL in some ways although they still had the ability to participate," admitted Lord Birt. Rumours of a lawsuit never materialised. The fine print in Mayhew’s contract with the computer giant kept the BBC in the clear. (Beeb.com would eventually be repositioned as a shopping portal before it was closed in 2002.)
The BBC’s R&D wizard Brandon Butterworth – “another hero”, according to Lord Birt – “was someone who had all the equipment needed in his bedroom”. Butterworth had already put up the first website from Kingswood Warren, the sprawling Surrey mansion that housed the BBC's research team. And the BBC had even briefly operated an internet service provider called the BBC Networking Club.
BBC Online, separate from the news website project, was established as an experimental area. After some investment, it fell under the control of former Tomorrow’s World producer Ed Briffa, who reported to Broadcast – the TV and radio empire that was historically at loggerheads with the BBC’s news departments. BBC Online had little clear vision of what it should do – but then few people in the mid-1990s did.
Some 12 years before he became a life peer, Lord Birt joined the BBC in 1987 as deputy director general. He arrived from ITV with a brief to sort out news and current affairs, and hand-picked two journalists in their 30s – Tony Hall and Jenny Abramsky – as lieutenants to lead his revolution. A decade on, Hall was director of news, and Abramsky head of the awkwardly-named "continuous news".
Bob Eggington
Abramsky sounded out Bob Eggington, a tough and practical editor, described in Lord Birt’s memoirs The Harder Path as a “dedicated, driven and no nonsense journalist” to head up the news website project.
Eggington wasn’t sure about it. He knew the corporation was on rocky ground – but he knew that if the operation could be built quickly and did a good job, its momentum might overcome corporate inertia and win over doubters. He secured a direct line of command to Abramsky and Hall – and a promise to shield the operation from the BBC’s bureaucracy. This would prove invaluable.
Mike Smartt joined as the news site’s editor. He had been a BBC TV news correspondent at home and abroad – and was an early technology adopter. He’d demonstrated to management that he could file scripts on a laptop “without harming himself or anyone else”. Smartt was fresh from the BBC’s 1997 Election website. For Eggington, Smartt “had thought about doing a news website longer than anyone else – and knew more about it”.
Lord Birt was famous for insisting that vast, deeply analytical reports are produced before any decision could be justified. When it came to the internet, he made the decision to ramp up online activity, then commissioned the reports later – and he was right. As one BBC insider put it: “Birt saw that something needed doing, so he set up something with the freedom to do it."
Eggington told The Reg: “He has his detractors, and sometimes I'm one of them, but the whole BBC internet effort happened because of John Birt.”
The peer is more modest today and deflects credit onto others.
“The BBC is a delightfully creative organisation: whether it's well managed or badly managed, these people will express themselves,” he said. “People were broadly alert to the internet. But they can't coalesce, so to speak, unless the management stands behind it and invests in it.
“We didn't act perfectly on every insight we had – but the news demo was the critical moment – I knew this is going to be an important public service medium, and it has to be funded by the licence fee, and it's going to be very successful. And we've got to be there, and be pioneers.”
There was a snag, however. The BBC wasn’t allowed to put up web pages. Nothing in the royal charter – which sets out the public-funded organisation's purposes – permitted it to do so, and the Ministry of Fun was ready to pounce on any transgression. The lacuna threatened to paralyse the BBC’s operation. Lord Birt then did something quite bold: he gave the order to proceed with a news website without the ministry’s permission to do so. A formal approach would have taken months.
This meant everything about the news website had to be provisional, like some kind of skunkworks project; until the BBC could secure an agreement, bureaucrats in Whitehall could have pulled the plug at any moment.
It was a genuine concern. Lord Birt recalled how the BBC’s attempts to put the World Service online had been abruptly halted when the British Foreign and Commonwealth Office, which sinks public cash into the international service, simply struck the line out of the budget.
“It wasn't a paralysing uncertainty. I assumed the government may fire a shot across our bows – but are they going to stand in the way of something that the BBC has a deep conviction about?" said the peer. "I'm sure I didn't have a deep worry. It was an irritation rather than a concern.”
It was now May 1997 and Eggington had no staff and no plan, but he had a deadline. He did hear some good news: the team leader at a major news rival was considering joining – and even better, may be able to bring some of his people with him.
COMMENTS
bonus tracks (tech speak alert)
After reading this, it occurred to me that I should have name-checked more people when I was interviewed. It was an incredible team, so here are a few more tales.
One of Jason Birch's many contributions was the said "FTP script" which replaced MS Replication Server. It wasn't so much a script as a highly scalable, fault-tolerant replication system (written in 36 hours), a direct descendent of which is probably still running today.
Matthew Wood was the chap who turned us all on to WebObjects. Convincing me really took some doing, because we didn't have the server muscle to use it in its intended dynamic mode. Instead, we wrote a system to capture dynamically published pages as files, then push them out. This was fantastically cheap, because Apple's licences were based on the number of simultaneous connections to the server. Using dynamic serving, there would have been thousands of connections. Using our system, there were only six.
Antony Tittle had the thankless task of dealing with the sports, weather and financial feeds. All the providers would make unannounced and undocumented changes to their formats, usually during the wee small hours of a New Year's morning. Ant would remotely log in and tweak the parsers, probably without putting down his pint.
Our original sys admin, Chris Hughes, personally attended every server crash within minutes, day or night, with a level of discipline which almost compensated for the anarchy reigning throughout the dev team. His first move was to eliminate novelty server names. He was also the only person who shared Jason's surprising taste for European dance music - Encore Une Fois!
Definitely a success
There are a vast number of reasons you could slag the BBC, but news.bbc.co.uk, or its equivalent, has been my homepage on every system - even my mobile - for well over a decade.
Simple design, easy to navigate, not too loaded with scripts.
Now, if they could only extend the HTML5 video to encompass more devices, I'd be very happy
BBC on the net before 1996
For the record, Brandon Butterworth did many important things during this period, but he did not create the BBC Networking Club, in fact he had nothing to do with it and didn't like it. I should know, I was involved in setting it up. In April 1994, the BBCNC put up the BBC's first Web pages in conjunction with a programme on BBC 2 called The Net. To the best of our knowledge this was a first anywhere in the world.
At that time, the Reith IP network in the BBC was the private domain of BBC engineers and jealously guarded by them. They had a 64kbps connection to the outside world and didn't want anyone else using it for fear it would use up their bandwidth. They never went anywhere near programme makers. So, when we set up the BBCNC as an initiative of BBC Education (as it was called at that time) we had to go outside the BBC to get server support from a small company called TECC, persuaded Pipex to expand their dial-up network, and lobby BT to get a 1Mbps line into their offices in time for the first episode on April 13th. We also had to battle with policy planning to persuade them that the Internet was a tool of the CIA and filled with porn (they had a point). I can remember going to see Brandon in an engineering building out in Acton somewhere, sitting in a darkened room, and having the strong impression that he didn't really want to help us at all. Anyway, we did start inviting him to meetings because BBC staff started wanting Internet access and were using the BBCNC dial up software to get connected, which was daft, so we were trying to help him put pressure on his bosses to expand their connection to the outside world and help BBC staff get access to the Internet directly. It wasn't easy. Meanwhile, we travelled all over the place, from BBC Wales to BBC Scotland, to the Worldservice, to Bush House and to BBC News -- I remember quite a few sessions with Mike Smartt. We were picking up work putting all sorts of very basic Web pages together for programmes, and from time to time doing specials like Radio 1's first Internet night in March 1995.
While doing this, we were constantly fighting off BBC Worldwidewho believed that that the only thing to do was a deal with Microsoft, which the McKinsey consultants involved were backing, and building a concensus that the BBC should consider the Internet a public communication channel, rather than a new commercial opportunity. That didn't stop a deal being done eventually with Fujitsu ICL which had doom written all over it from the start. But the BBCNC was canned to make way for this deal, on what turned out to be somewhat spurious grounds that we had crossed the line between public service and commercial interests, but there we go. The BBCNC team got merged into a BBC Multimedia Centre during 1996, which in turn got overtaken by BBC On-line.
Anyway, got that off my chest. As I say Brandon did a lot of important work, but it was the BBCNC that got him moving and the BBC's presence on the Internet into the sunshine, not the other way round, and no matter what his blog piece says!!!!
IT infrastructure monitoring strategies
What you need to know about cloud backup
Enabling efficient data center monitoring
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
