Accenture in doghouse after NHSmail mass outage cuts off 1m+ UK health staff

Accenture is reportedly facing major financial penalties after a failed upgrade took down the email system used by about 1.2 million staffers at the UK's National Health Service this weekend.

NHSmail – delivered by integrator Accenture since 2015 – suffered widespread outages on Saturday, described by NHS chief clinical information officer as “exceptionally unusual”.

NHSmail is completely down on all platforms. This is exceptionally unusual. The team and supplier are investigating. Updates via @NHSDigital https://t.co/Olh1Yuw6EA

— Simon Eccles (@NHSCCIO) December 1, 2018

Users were locked out of the NHSmail portal, email in Outlook or Outlook Web Access from about 11am on the morning of 1 December. It took about six hours to resolve the issue, and access wasn’t fully restored until about 8pm that evening.

In a statement, NHS Digital service director for live services, Neil Bennett, said the group didn't yet know how many users were affected, but that it would be a "significant number".

He added that early investigations suggested the problem "was caused by a fault in a third-party hardware device" – and it is Accenture in the firing line, with reports that it upgraded the data centre hosting the email servers last month.

According to the Health Service Journal, which cited sources close to the matter, the firm is now facing “big penalties” for the shutdown, as it had failed to meet its contractural requirements.

HSJ reported that not only did Accenture’s testing fail to detect the error in the upgrade, backup systems also failed.

It said the fault occurred in two separate data centres hosting the service that caused what the publication described as a ”memory leakage” over a period of weeks before the complete shutdown on Saturday.

Not all staff use NHSmail – some organisations use their own email systems – but NHS Digital pitches NHSmail as a more secure option that provides access to central services to organisations and users.

Meanwhile, NHS Digital recently announced that NHSmail is working with Microsoft to provide a hybrid version of Office 365 that integrates the NHSmail directory, exchange and Skype for Business services. The final version of the subscription-based cloud suite was due this autumn.

Accenture won a five-year, £60m contract to deliver NHSmail in 2015 – it had previously run on Microsoft Exchange 2007.

But the firm was in hot water just a year later over a mass reply-all email fail that saw 500 million emails sent in just 75 minutes, after a test message was sent to all 850,000 NHSmail account-holders rather than a local distribution list.

That was pinned on a "software configuration error", which meant that when the administrator selected "only in my organisation", the system applied an "All England" rule rather than just one for her organisation.

NHS Digital said in a report on the incident that Accenture had failed to deliver a design requirements for NHSmail that called for "strict controls… to limit the volume of any one email sent by an individual user or local administrator".

Accenture referred questions on the outage and possible fines to NHS Digital, which didn't comment on the potential of financial penalties.

NHS Digital said it is launching a post-incident review and would publish a report through its board. ®