Feeds

Judge cracks down on Bayesian stats dodginess in court

Terry Pratchett effect angers beak

Providing a secure and efficient Helpdesk

Analysis A judge in a (sadly unnamed) British case has decided that Bayes' Theorem - a formula used in court to calculate the odds of whodunnit - shouldn't be used in criminal trials.

Or at least, it shouldn't be relied upon as it has been in recent years: according to the judge, before any expert witness plugs data into the theorem to brief the jury on the likelihood that a defendant is guilty, the underlying statistics should be "firm" rather than rough estimates. The decision could affect things like the odds of matching drug traces, fibres from clothes and footprints to an alleged perp, although not DNA.

In a murder appeal case, brought after a man was convicted on the basis of his footwear almost matching a print linked to the crime, this precise point was made:

The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates.

Mathematically leaning Reg readers will be able to make much more sense of the details than I as a mere journo will be able to. But from a legal point of view this looks like a good ruling.

Yes, Bayes' Theorem can indeed be used most usefully to make estimates, which give us a good idea of what is likely to have happened. However, that's not quite the same as giving us the information leading to “beyond reasonable doubt” which is what we require before locking someone up.

More than that, the way that the statistics are presented can be more than a tad misleading. For a start, the jury is made up of the general population, not exactly a hotbed of sophisticated statistical reasoning, and being told by experts that there's a one in a million chance leads to an all too common error.

A DNA match to one in a million does not mean that it's a million to one against the bloke 'aving done it, m'lud. Rather, it means that in a population of 65 million that 65 people, based purely on the DNA, could have done it. Our DNA tests thus mean that we now have to go and exclude those other 65, or at least regard them as the prime pool of suspects, not convict our man in the dock purely on the basis that one in a million is beyond that reasonable doubt. Yes, these sorts of mistakes are made in the chain of reasoning.

It can get worse, of course - mentioning no names, no pack drill as it's still a case that gets people het up - the likelihood of any one child dying a cot death is 1 in 79,000 (entirely made up number for illustrative purposes). Two children from the same family dying of cot death is thus 1 in 79,000 x 79,000 which is 1 in 6,241,000,000. One in six billion, so, members of the jury, you know what to do: lock up the mum.

This was actually the logic used by one eminent expert witness: the appeal was eventually allowed, some years later, when it was pointed out that cot death might not actually be an independent event, that perhaps there is a genetic predisposition to it, perhaps the environment means that one cot death increases the chance of a second one. Given one cot death, the chance of a second might only be 1 in 1,000. Or 2,000 (again, made up numbers) which we most certainly wouldn't want to use as the basis of “beyond all reasonable doubt”.

Neither I, the judge, nor anyone else has any serious doubt about the usefulness of Bayesian reasoning in evaluating evidence in court cases. But the techniques have been so badly understood, even by experts, in recent years that a rethink, a stop and a reasoning through all of the implications, doesn't sound like a bad idea. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
WHY did Sunday Mirror stoop to slurping selfies for smut sting?
Tabloid splashes, MP resigns - but there's a BIG copyright issue here
Spies, avert eyes! Tim Berners-Lee demands a UK digital bill of rights
Lobbies tetchy MPs 'to end indiscriminate online surveillance'
How the FLAC do I tell MP3s from lossless audio?
Can you hear the difference? Can anyone?
Inequality increasing? BOLLOCKS! You heard me: 'Screw the 1%'
There's morality and then there's economics ...
Google hits back at 'Dear Rupert' over search dominance claims
Choc Factory sniffs: 'We're not pirate-lovers - also, you publish The Sun'
EU to accuse Ireland of giving Apple an overly peachy tax deal – report
Probe expected to say single-digit rate was unlawful
While you queued for an iPhone 6, Apple's Cook sold shares worth $35m
Right before the stock took a 3.8% dive amid bent and broken mobe drama
prev story

Whitepapers

A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.