'Incomprehensible failure' – Canada's $1bn Phoenix payroll IT fiasco torched by auditors

Govt tech nightmare caused by bad bosses, poor planning

A tire fire

Canada's top auditor has issued a scathing postmortem report on Phoenix: the nation's disastrous attempt to overhaul a key government IT system.

In its comprehensive dossier outlining the nine-year-long fiasco to replace the software that handled the country's payroll system for government workers, the Office of the Auditor General slammed the decision-making that ended up underpaying thousands of employees.

"We concluded that the Phoenix project was an incomprehensible failure of project management and oversight," the report reads. "Phoenix executives prioritized certain aspects, such as schedule and budget, over other critical ones, such as functionality and security."

Unveiled in 2009, the Phoenix platform was pitched as a complete replacement to the 40-year-old payment system used at the time by the Great White North's vast array of government services. IBM was chosen as the contractor to deliver a system based on Oracle's PeopleSoft system.

What started as a CAN$310m project would balloon by a further CAN$540m, and broke down soon after it went live in 2016. That's CAN$850m total, or $650m (£500m). The Trudeau government estimated the total cost to fix the system could hit CAN$1bn by 2022.

UK Home Office's £885m crim records digi effort: A 'masterclass in incompetence'

READ MORE

While the project involved products and services from both Oracle and IBM, blame for the Phoenix failure was placed squarely on the shoulders of Canadian government officials, specifically one department: Public Services and Procurement Canada.

"The Department did not completely and properly test Phoenix before its implementation, which is contrary to recognized practices for developing a system," the audit stated. "Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide."

From the start, we're told, the IT project was fraught with management and delegation problems. Among them, basic errors such as the inability to handle requests for retroactive payments. The requests, the auditor general said, were more or less ignored by a group more concerned with simply getting the system online.

"The Department knew about many of these critical weaknesses before implementing the Phoenix system," the report reads. "In our opinion, these weaknesses were serious enough that the system should not have been implemented."

Recommendations

Looking ahead, the auditor general is recommending a set of new requirements for how the Canadian government should handle major IT projects, including the use of internal audits and oversight groups that would help keep a close eye on how projects are being rolled out. The report also called on the department to bring in someone to define and assign roles to everyone involved in a project.

"The department will ensure that project managers understand and respect these requirements," the report stated.

"It will also ensure that project managers understand well the requirements for preparing a stakeholder engagement plan, a formal software upgrade plan, and a comprehensive contingency plan that covers government-wide systems, processes, and impacts for all government-wide IT projects under its responsibility."

While specific to Canada's Public Services and Procurement department, the findings should provide an outline to all government and large organizations on the dangers of undertaking major technology projects without properly planning deployment and setting up plans for regularly auditing projects and responding when things go wrong.

Hopefully, the people behind the next Phoenix-style project will take the lessons to heart, rise from the ashes rather than set everything on fire, and another costly disaster can be avoided. ®




Biting the hand that feeds IT © 1998–2018