UK.gov IT: it's broke, how can we fix it?

Letters This week's analysis Building disaster into the network: how UK.gov does IT drew a hearteningly substantial and thoughtful mailbag. It's always nice to be agreed with, but my point that the British Government in particular is designing failure into its IT projects and infrastructure clearly caught the mood of the team. Various issues follow from this, with the letters handily falling into three categories.

First, if we can't do large scale IT projects that work, then we will be in big trouble as we attempt to implement specific projects such as the ID scheme, the NHS IT programme and the benefits system (ah yes, we just did that, but it's not finished yet). Second, blame the contractor; yes, up to a point, and contractors who make catastrophic mistakes should be spanked. But if you don't understand what they're doing, this will happen with any contractor. Sack 'em if they're crap, but have sufficient knowledge and supervision to find out they're crap before they crap on you. Finally, most of the longer contributions deal with the broader issues of the deskilling and disempowerment of government IT and the consequences. A couple of the contributors either are or have been fairly senior, so have direct knowledge of the process.

One last point for you to consider before we get on with it. Yesterday LibDem leader Charles Kennedy suggested to Tony Blair that he should bar companies from bidding for the ID card scheme if they had been responsible for failed government projects. I'd like to suggest that he has the wrong target here, because if a project is badly conceived and specified by the customer, it will always fail. Blame the contractor, sure, but (sympathy for EDS) if you go to the contractor with something vague, then continually change your mind as the project goes ahead, your costs will climb, it'll be late and it won't work. So I think we should bar ourselves from specifying an ID scheme until UK.gov has passed its IT proficiency test. The opposition parties are slightly helpful in moaning about the cost of disasters, but if they get into government and haven't figured out the real problem, they'll just screw up too.

On with the mailbag:

What if this happened to the "ID card network" that is currently being proposed/suggested/implemented. If ID cards are going to be a pre-requisite for all the things I've heard rumours about the whole country would come to a stop
rgds Roger

It certainly makes me wonder about the feasibility of other, looming IT ventures like those for the NHS and Royal Navy. Questions like, will these be outsourced too ? Answers like, "Yes, to India or China" keep popping into my head.

I can only say that I'm glad the government has it's priorities right in testing out the calims of the consultants and developers on the sick, poor and needy of the UK just before Christmas, rather than waiting until some urgent global conflict requires a fleet of nuclear subs. To be sure, a week without most of our Navy, is a different matter to a week without food and drugs.
regards David

We've all been remarkably good in not mentioning the Beast of Redmond (although the Windows nature of the original story did produce a large bag of the usual 'there you go' emails), but there are a couple of issues here. Moving to a system of networked Windows PCs is now a pretty standard goal for UK Government departments, and the more critical these workstations become to the continued running of the operation, the more imperative it is that that the departments clean up their acts as regards security and discipline, and plan resilience into the network. The other point is the issue of where it could have been that these sloppy admin people got their slipshod notions from. I personally think a certain very rich person who does a lot of selling the joys of sharing and automation might have a certain amount to do with it. But that's just my opinion.

"DWP should have been in the position to know and understand why the button could not have been pressed"

-Should it? I would have thought this is exactly the sort of stuff that you decide isn't a "core competence" and farm out to an expert contractor. Seriously, I think this should be landed fairly and squarely at EDS' door - bunch of muppets!

Still, no harm done eh? Luckily the only people affected were society's vulnerable - and lets face it they are used to it right? Besides, they tend to be too immobile or homeless or lazy to vote.

Mark

Curiously, Mark is not the only one to use the word "muppets" - is this internal DWP lingo?

Peel Park, Blackpool, probably where the incident started is a babble of different teams - many who don't even talk to each other!

quick example is the server team - who are EDS staff, but the routers are controlled by an other company - Syntegra. To complicate it even more, if a serious problem is logged, each team has different logging tools - ie. if the server team views an incident in their incident tool (peregrine) and it's a network fault, then Syntegra team has to wait for the incident to be "converted" to their logging tool IMS - and vice versa. To make it laughable even more, both teams reside at Peel Park! I think they are in the same section as well (?).

DOI as you mentioned was originally called EOI (Early Office Infrastructure). Does it take a great mind to realise that "early" would soon become outdated? It did, and so DWP - EDS had to change loads of EOI to DOI items (paper, applications...) costing - hell knows??

DOI is web based and not so bad as it once was. It's still classified by CSO's (computer support officers) as crap! Did you know that trained civil servents (CSO) are so limited they are not allowed to "ping" their own staff's workstations to test if a network card is faulty in that PC? What happens? It gets sent to the EDS-RMC (who manages IT incidents) as an urgent costly incident so a team can "ping" that PC and charge the Gov many lovely of ££££s to tell the CSO that the PC is broke.

This is all chit-chat, but it does go to show in a little way that DWP-EDS have severe problems. No thought is given to the long term - strange that EDS is also having problems in other "high profile" accounts as well ;-)
(sorry for being anonymous but I work for DWP-EDS - hopefully not for long!)

Starting to get a bit nasty there. But note the apparent absence of a coherent and logical structure. Presumably DWP once had one of these, but has mislaid it somewhere on the road to outsourcing. And nastier still coming up, with names removed in the interests of fairness and our legal costs:

Some little titbits for your article though you may already know them if you're already talking to IT Contractors at Peel Park.

XXX was the muppet who published the XP 'fixes' to the live network & he seems to be immune to all retribution though rumours abound of XXX & XXX [senior staff censored by Reg] taking a fall

All of the people in the ISA team, the Messaging Team (Exchange) & the Server teams have Enterprise Admin privileges on their accounts that they use daily.

All of the DCs are in only 2 locations despite there being several powerful servers on most sites & possibly the worst network links in history.

XXX [cut for legal reasons]

All of the servers excepting the DCs are still at 2000 Server SP2 & patches have been on hold for some time.

It's a mess......
Anon

Very wise of you, Anon. That looked like a Saudi copy of Playboy. We've passed the information on to a couple of people we hope will make a difference.

This was one of the best articles I have read on this subject. Your diagnosis of the problem is spot on. The IT systems that run our government are basically specified by civil servants who know little about technology and are implemented by private corporations that have no understanding of the needs of public administration. I think the politicians and senior mandarins do not care if the suppliers can deliver just so long as there is a name in the frame to take the blame when things go wrong. Unbelievably. many of the systems implemented now are actually inferior and more expensive than those created when the job was done within the civil service by its own cadre of trained technicians and project managers. The idea that outsourcing was necessary to gain the technological expertise required to develop modern applications is simply rubbish. Any one who worked in government IT in the 1980's will know that private sector consultants were regularly employed to provide expertise when needed. Moreover, the current set of contracts to deliver government IT are so complicated that there are now more people employed administering them both within the civil service and the private suppliers than there are actually working on the design and implementation of new applications. At the end of the day the real victim of this farago is the taxpayer who ends up footing the bill whilst receiving inferior services.

Mark

I worked in the in-house IT Team for a division of another government department (hello trees, hello flowers, hello agrobusiness), until the brains-in-jars decided to follow the management group-think and outsource to the private sector, at which point I moved sideways sharpish. "You want me to work for X, with my current non-contrib pension replaced by one that is "broadly comparable", with the inevitable rationalisation once X knows how may people ACTUALLY work, and still with the same jarheads managing the network? Or leave it to the "professionals" with an MCSE (Microsoft Continue Shovelling Excrement) but without my experience, and do a desk job? Oh dear, I'll have to think about this. I've thought about it. Ta Ta IT!".

It's all about reducing headcount in the Civil Service, and if you can get someone else to take the responsibility when the systems fail, then that's just grand! I seem to recall though that when you "empower" someone else to do things for you, you still retain the responsibility for delivering the results!

Still, I sleep better at night now knowing that when the inevitable twit infects the network with a file from home, bypassing the (tinfoil hat)Intranet Asegura del Gobernmiento (/tinfoil hat) and poisoning all our XP boxen, I don't have to move a muscle. I can leave it to HAL. It'll only take a day or so to fix, won't it?
anon

If we recognise that department right, then the lucky outsourcee is IBM, right? We mention this in the interests of balance, and as Thought for the Day might say, are we not all, in some way, guilty?

You are half right.

First, the process of outsourcing has removed the capacity of government departments to appreciate what their contractors are doing. When the late 1990s outsourcing contracts (usually for about 10 years - so you know what network, OS, desktop and communications requirements are going to be in ten years' time) were set up, the in-house techies typically went with them. So we now have incompetent senior managers negotiating complex long-term outsourcing contracts they do not have the qualified staff to monitor and enforce. But by then they will have moved on.

Second, as a product of the first, the framework contract tightly specifies a minimum, fixed set of deliverables (3 hardware upgrades, four software upgrades, etc, for x desks in y locations, with interconnections at z levels of broadband, etc), not bad for star gazing in year minus two, but not in the real IT world. So everything the world brings, and everything Ministers and civil servants legitimately want to change is a variation of the contract, outside the nice tight price ceiling bid that won the stripped out, and totally unrealistic competition. Even if the initial build can cope, every line of code costs..

Screwed by real life then. But also by a specification process which is typically also courting disaster. The normal request is "automate this" where "this" is a paper based, multilayered, audit proof system, rather than "help me deliver this service". And typically at the specifcation stage the "requirement" is "collected" from "stakeholders", rather than designed to maximise the benefits of IT management, even if that means some stakeholders (= current manual handlers) losing their functions to a database. So from the first stage management abdicates its responsibility, before it hands over the expertise to manage the outcome.

This incidentally is not an accident. The approved methodology for Whitehall projects, including IT, is called PRINCE 2. It's available publicly. read it and tell me who ends up with responsibility for anything. The latest improvement has been to introduce Cabinet Office/Office of Government Procurement monitoring and "gateway assessment", which only further dilutes direct responsibility, and leads, on past form, to the blind monitoring the sightless.

If it's not an accident or a cock-up, is it a conspiracy? Possibly. The last politician to take the fall for a departmental mistake was Lord Carrington (remember the Falklands). Why should civil servants expose themselves to career killing risks for their ministers, if the latter will refuse to take responsibility for, eg the Prison Service, failing to meet school performance standards, DWP delivery systems, etc? We know from "Yes Minister" that the civiil servants run rings around the politicians, so draw your own conclusions. At least don't assume they are stupid.

Thirdly, the senior UK civil service believes, probably rightly, that individuals need to move onwards (and upwards) usually every two to three years: they are anonymous - policies should be identified with politicians. It keeps them intellectually fresh and challenged. But where has an IT based transformation project in a multi-K staffed organisation spead over possibly hundreds of locations been deliverable from intial qualification of suppliers through detailed specification and contract procedures to operational delivery in less than three years? IT failures are department wrecking events, but there is never a senior identifiable civil servant, or minister, around long enough to be nailed.

But wait a minute. Who ended up being held resposible for the "intelligence" failures of earlier this year? Ministers? Senior civil servants? Spooks? Sorry, all nicely dispersed through committees. IT failures are a subset of a much wider malaise in the British political and bureaucratic structure. Show me the buck.
Anonymous - please respect.

Good treatment of a very important subject. Unfortunately, I doubt that the PHB's in your government's IT establishment will take heed, as you so astutely predict. Whether in government or private business, many of those charged with making IT policy seem hell bent on taking their organizations over the same cliff. As is reflected (unintentionally) in an article in a recent Information Week, they continue to cling to outsourcing as a solution even in the face of their own disatisfaction with the results. My own experience with outsourcing companies, including the giants like IBM and EDS, is that they are by and large focused on increasing their own revenue without regard to actually delivering results. Too often the mantra that a particular issue is "outside the scope" and avoidance of responsibility for thier own incompetence has gone without challenge by supposedly "tough minded" and "visionary" IT managers.
Phil

Phil is sensible to note that this is not just a government problem. There are certainly characteristics of government that tend to accentuate and foreground the problems, but losing control of your own IT is an issue all businesses should be worrying about. We had many more emails, so apologies to those not published, no slight intended. We had only one rude one, of the 'I don't expect you to publish this' variety. And we will not disappoint you, sir. But we do publish rude stuff, and in your case we just haven't published it because your claim is so weird. It was no big deal, and the staff only experienced some disruption? Er, sorry? ®