Understanding of ‘black box’ IT systems will reduce Post Office scandal-like risk

Another Post Office scandal could be avoided if leaders in public bodies understand the “black box” IT systems that run their organisations and encourage a “speak up” culture, according to a Parliamentary report.

In its latest report, Recognising and responding to early warning signs in public sector bodies, the Committee on Standards in Public Life cited the Post Office scandal, among others, to highlight failures in public bodies.

In his forward, committee chair Doug Chalmers, a former British Army officer, said the Post Office Horizon, Grenfell, Windrush and infected blood scandals are “very different in nature” but all had “a catastrophic impact on human lives”.

“It isn’t hard to find common themes among these scandals – a failure to listen to and act on concerns raised, a failure to learn lessons from similar incidents, and a failure to identify and share emerging risks,” he wrote.

The Post Office scandal was fuelled by all of these failures and more. The Post Office management ignored subpostmaster pleas that the Horizon IT system was causing unexplained account shortfalls and failed to investigate them, choosing to blame the subpostmasters for discrepancies that didn’t exist outside the IT system.

The failures went way beyond the Post Office itself, with its government owner neglectful of the so-called “arms-length” body. Meanwhile, Fujitsu, the supplier of the controversial IT system, made the Post Office aware of problems with Horizon but did not make them public. The supplier’s staff even gave evidence during the trials of subpostmasters, who were charged with crimes of dishonesty, where they wrongly stated that Horizon could not have been responsible for the unexplained shortfalls in branch accounts.

The Committee on Standards in Public Life report said “black box” systems like Horizon, which not only ran the accounts of thousands of Post Office branches but also provided data to prosecute people, must be understood by leadership teams.

Systems like Horizon are described as black box because it is clear what is input and output, but not the workings in between.

“Leaders of organisations that use ‘black box’ systems should be asking themselves whether they are confident that they have sufficient understanding and oversight of how these systems operate or whether they need greater assurance about their use,” said the report.

Beyond the system itself, the report said people need to be empowered to “speak up” when they see failures. During the Horizon scandal, which began when the system was introduced in 1999/2000, dissenting voices were silenced and the Post Office managed to keep a lid on talk about Horizon problems until 2009, when Computer Weekly helped campaigning former subpostmasters make the Horizon problems public. In that time, huge suffering had been inflicted on subpostmasters, who were blamed and punished for unexplained accounting errors, including hundreds being wrongfully imprisoned.

Beyond the human suffering, the scandal, which could have been prevented following warnings in the late 1990s, is set to cost UK taxpayers billions of pounds.

The report foreword advised on what organisations can do to increase the “likelihood of risks and issues being uncovered”.

“Culture and leadership, at all levels, are central to ensuring that these processes are effective. And that building an organisation where it is second nature for people to speak up about concerns is an art and not a science,” it stated.

“It is not always easy to speak up – it requires moral courage to be the person who says, ‘I’m not sure this is going to plan’ or, ‘Is there a risk that if we do X, it will have these negative consequences?’”

According to Neil Gordon, a professor in computer science at Hull University and chair of the British Computer Societies Ethics group, the report also makes interesting reading for computing professionals beyond the public sector. “As professionals, we should be acutely aware of the impact of systems, whether safety-critical or apparently more mundane, such as accounting software.”

He added: “There is a need for those providing and supporting such systems to make sure our customers and users appreciate their limitations and deficiencies. Furthermore, this illustrates the need for all organisations – public or not – to consider their mechanisms for identifying risk and harm, and encouraging open dialogue with employees and others to address them.”

Gordon said IT experts can play a pivotal role in preventing organisational failure by analysing data to identify risks as early as possible and help in decision-making.

“Artificial intelligence [AI] may be an effective way to do that, provided the systems are themselves developed appropriately. Emerging technologies – from AI to quantum – will create new opportunities to promote human welfare, but equally, they can do harm,” he told Computer Weekly.

“Whilst the report presents a strong way forward, the need for different mechanisms – whistleblowing and scrutiny by the press – remains and we welcome progress of support for those who do raise valid concerns,” added Gordon. “This also highlights the importance of codes of conduct and that we all have a duty to take on responsibility so we can reduce the likelihood of the sorts of historical failures described in the report, and to minimise the damage where they occur by identifying the problems early and raising the alert.”

A Post Office spokesperson said: “We will examine the report and any learnings in detail. The Post Office has made a number of cultural changes in recent years, including the appointment of serving postmasters to the board, and we operate a ‘speak up’ whistleblowing service enabling our employees and postmasters to raise concerns in confidence and anonymously if preferred.”

Computer Weekly first exposed the scandal in 2009, revealing the stories of seven subpostmasters and the problems they suffered due to Horizon accounting software, which led to the most widespread miscarriage of justice in British history (see below timeline of Computer Weekly articles about the scandal since 2009).

Read more on IT project management

• 

  Kroll reviewing Post Office Horizon’s current integrity and discrepancy identification

  

  By: Karl Flinders

• 

  Post Office scandal data leak interim compensation offers made

  

  By: Karl Flinders

• 

  Government announcement on Fujitsu talks add ‘vague words’ and no interim payment

  

  By: Karl Flinders

• 

  Metropolitan Police concern puts brakes on Post Office Horizon data migration

  

  By: Karl Flinders