Original URL: https://www.theregister.com/2012/07/23/olympics_cio/
Is running IT for the Olympics the worst job in the world?
The results will be in in August
Posted in On-Prem, 23rd July 2012 09:03 GMT
It's going to be the Worst Tech Job in the world ... or the best. It all depends on how the Olympics IT works this summer.
Michele Hyron will be either in front of a Parliamentary committee explaining why the tech at the Olympics was a fiasco, or enjoying a well-earned rest - because she is ATOS's chief integrator for the Games in London this summer.
Michele Hyron: Calmer than most of us would be...
Earlier this year, I trekked out to meet her at the secret location where ATOS will be managing the Games. Hyron has a unique CV, being the only Olympic IT project manager who has worked on this stuff since 1992 – which will look good if the Games' tech works. But if it all goes belly-up, she will be in rather less demand.
SysAdmin as a spectator sport
You’ve had management peering over your shoulder and maybe thought you had it pretty tough, but the BOFHs at the Network Operations Centre are putting up with a spectator gallery built over their area so that random busybodies (like me) can be entertained by the sight of people staring at screens, a spectator sport to compete with golf.
In keeping with the sporting theme, the gallery has astroturf on the floor instead of carpet – of a strange, unlikely green hue. Big screens are mounted at the front in NASA control configuration and it’s all unnaturally tidy for a space that will hold 180 techies at a time, 24/7.
There is distinct tension in the air when I visit, with people talking of the number of Mondays left to go. The team has been built almost from scratch for 2012, but come the summer, it will have upwards of 5,000 tech staff trying to hold things together, mostly at the 94 venues.
All this effort is expended on behalf of Atos, which is apparently Europe’s second-largest IT services firm. I suppose I ought to have known that already, but I’ll bet you didn’t either. It does corporate IT and has no consumer products, so the Olympics are a showcase for its duller work of programming, project management, system integration et al. The payback is showing people round the main venue while showing off the fact that it has actually managed a major government-linked project without going belly-up (at the time I write this, not a forecast). Atos is ferrying IT execs round in batches, so if you’re offended that someone at your senior level hasn’t got an invite, let me know and I’ll pass it on.
The one thing you must know
If you’ve met people from Atos you already know that it doesn’t manage the ticketing technology, often because it is the first thing the Atos techies tell you. This is probably wise, given the spectacular mess that Ticketmaster has created and the odd way that tickets are allocated in general. Instead, Atos' job is to get the many diverse bits of IT working together securely under a serious media spotlight – which will only increase as 27,000 of them will be there, plus a billion or four watching. Hyron seemed surprisingly relaxed given how public any failure will be, especially after what happened to IBM ... but more on that later.
Guy Mowbray doesn't know everything: The touchscreen
tech that gives commentators something to prattle on about...
The most important Olympic Games technology that you will never see is the commentator information system. Being a Reg reader, you probably worked out that the aged jocks who witter about it being “the third-fastest time for a female Belgian 400-metre runner this year” aren’t blessed with photographic memories but a mix of real-time and static data on their screens.
At least that is the theory
Until the Atlanta Games, IBM did all the IT for the Olympics, but the Commentator Information System went titsup – annoying a large (even by IBM’s standards) number of the media. IBM doesn’t do the Olympics anymore and its decline from being the world’s largest IT firm started about then. (Though it may be because the firm decided to focus on higher margin services and software revenue – that’s what IBM would say anyway.)
So it’s no shock Hyron is big on change control as well as processes and there is no tolerance of “it’ll be all right on the night”, being structured on an ITIL best methodology. The goal of “flawless” delivery means that for 2012, Atos is taking a highly conservative approach, upgrading existing systems rather than deciding to take the risk that cloud computing would be up for the job. The firm has also managed to ditch the Oracle database, standardising on MS SQL Server with Java front ends.
Cats in a bag
BT is providing the connectivity, except for the bits done by Cisco or Aircall. Acer is supplying the PCs and some of the servers, the rest being supplied by Sun.
Omega has the job of capturing results and applying the obscurely complex rules around each event, using power from EDF, and putting it on to a PC supplied by Acer but managed by Atos, connected by BT, through Cisco routers, onto the Sun/Acer server running both Windows and Solaris, back through BT/Cisco, onto screens supplied by Panasonic (who also watch you through the security cameras) and out through BT to the IOC’s tame vendors providing the general public with web information. Whew...
But it’s actually more complex than that
To add to the normal challenge of being a CIO, few of these suppliers actually have contracts with Atos or even are chosen by it. They cannot be fired. Unless, like Nortel, they go bust. Instead each has done deals with LOCOG and the IOC where a mix of money, goods and services are swapped for the chance to show off to a global audience. So Hyron can’t allocate work based on competence, nor has she the same vocabulary of threats I would want to use on any potential misbehaving vendors.
However she’s been doing this since 1992, requiring a mix of diplomatic skills that would make bringing peace to Palestine look easy given how badly any of us have seen some IT suppliers behave in terms of blaming each other or claiming that basic system documentation was a secret. One thing I respected about Hyron was her ability to project competence and confidence that it would all work.
So if I didn’t know that government IT systems were maliciously bad, the fact her systems must work with people like the Border Agency would still be pretty daunting. The Border Agency – incidentally, another Atos customer – will be trying to keep bad people from getting accreditation and there are security agencies from any number of countries to keep sweet.
One of the biggest changes will hopefully be invisible since they’ve replaced a mess of ancient ad-hoc protocols for transmitting games data to partners and national Olympic committees with a single XML-based data stream, so that they can configure real-time results and stats onto their own sites.
'Um, boss. I think I just deleted Athletics....'
Info+ - gives accredited media, officials and athletes access to info on schedules, rankings, weather, records and athlete biographies.
Your outfit is probably just coming to terms with supporting consumer tech like smartphones and tablets brought in by users. Atos is going to have 27,000 journalists turning up with pretty much every device known to man, configured in ways that would make hardened sysadmins cry, all while preserving security and not letting them accidentally trash each other. Given that from their perspective this is a marketing event, they really don’t want that many media people writing about system failures stopping them delivering copy.
Part of this requires BT to make a truly enormous Wi-Fi hotspot array work first time, Personally I’ve never seen big Wi-Fi work in any useful way, even the second or third time. Hyron’s team are also providing a drop-in facility to resolve the IT problems faced by 27,000 media people far from home.
British PFYs and BOFHs
Nearly all the technical staff are British, since apart from the Barcelona software development team, only eight ATOS staff are carried over from Beijing, along with 70 student interns and half a dozen trainees. Internships are routinely abused by some firms as a way of not paying for entry-level labour, but Atos is actually doing the decent thing and paying undergrads from places like Queen Mary's College just down the road.
Security
Atos is collaborating with the Border Agency on accrediting the 70,000 volunteers, 27,000 media, sponsors, etc as well. So out of the total of 250,000 people with some form of backstage pass, only 4,200 are actually athletes. That horde means there’s upwards of a thousand scanners and PCs to install and support and any number of people trying to fake tickets, crack the systems or generally cause trouble.
The accreditations are mildly biometric, so just scanning your mate’s shouldn’t work, but until the day no one will really know. There is a disaster recovery centre, which unlike the main secret control centre doesn’t have a big sign announcing its presence. Hyron’s team have actually run events in London already without Ticketmaster-level issues, but The Reg will be there to diagnose the mess if and when it goes wrong. ®