FBI's terror trawl and Emergent computer goofs
30 degrees of Bacon sandwich
Letters Our story analyzing why technology let the FBI down - Emergent cheese-sandwich detector enlisted in War on Terror - with catastrophic results, drew an impressive mailbag. If you recall, the Spanish authorities found a fingerprinted bag full of explosives a week before the Madrid bombings, and the FBI was convinced it had their Man. They had the wrong man - but a combination of faith in their"social software" and poor quality digital fingerprint led them to the wrong conclusion.
Such mistakes are expensive. A senior business analyst at a multinational has the most succinct summary of possible remedies.
The short answer is, by promoting critical thinking and a higher level of education and understanding in the young. There are several reasons why this will not, and possibly *cannot* happen:
1) People are lazy - they enjoying believing in magic wands that will make their troubles go away.
2) Science is the new religion of the 21st century. Things are taken on faith, rather than questioned.
3) There is an increasing tendency in the west to cling to authority. This manifests as the incremental growth of "nanny state" policies, as "experts" being trotted out to make definitive pronouncements in the media, and even as the recent oversimplification of the complex issues during the Iraq situation.
4) Our consumer culture reinforces all these tendencies, with phrases like "new"; "improved"; or "contains Natural Atropa Belladona" being seen as marketing tools. Very few consumers ask themselves what was wrong with the last version, why this one should be any better, or indeed, what the advantage is in having a poisonous plant in your breakfast cereal.
One of the chief barriers in my work is trying to make people understand that the all-singing-all-dancing system being implemented actually (gasp!) needs someone to *USE* it.
Another is the increasing tendency of senior managers to require that I motivate proposed solutions to complex issues with a "one-pager". And for some reason pointing to the large red six-digit figure at the bottom of the P&L is *too* simple...:(
I suspect that this is nothing new in the world, however. I have a mental image of Ugg the caveman cursing and swearing while a cave bear chews on his leg, and wondering why the Mark VI FlintAxe he got from Ogg the knapper yesterday is not beating the bear to death by itself. After all, Ogg said that it was a major improvement over the PointyStick Mark II...
Given that it is unlikely the root causes of this foolishness can be fixed, the only possibility is to demand tangible output and product, and not accept people or institutions passing blame over to some electronic scape-goat.
- Hold CIO's to account when their expensive SCM/ERP system fails to produce benefits. (fire them, with no golden handshake)
- Do not accept it when customer-serviceless drone says "the system made an error" (NO, a PERSON made the error ON the system)
- Do not accept apologies from organisations who blame their electronic tools. ("My hammer just smashed all your kitchen appliances. Maybe it got a virus?" would not be acceptable from a plumber working on my kitchen sink)
"To err is human, but to really foul things up requires a computer." might be a twentieth century version of Alexander Popes "To err is human, to forgive Divine" but I personally vote for a little less forgiveness, and a little more responsibility when things go tits-up.
[name and address withheld]
No-one with a good understanding of computers would have accepted the output from this pattern-matching program as blindly as these (presumably intelligent) FBI agents did. As you point out, a few minutes of checking the results would have revealed that the suspect couldn't have been on the scene when the evidence was found.
I've seen similar situations in business. To take one example, traders relying on Excel to analyse their positions in the markets (too complex for humans to evaluate). The dealers would probably spot any gross errors, but an answer that looks reasonable is accepted because "computers don't make mistakes". At least the result of an error would only be the potential loss of money and not loss of life.
Without wishing to appear too chauvinistic, I have to say that I've found the attitude that all problems can be fixed by the application of sufficient computer power (and money) to be more prevalent in the US than on this side of the Atlantic. I fear that I can already see signs that we're falling victim to this fallacy too.
Brian C Miller meanwhile objects to our description of agent software as "simply rubbing up against your leg, like a friendly kitten."
No, it's humping your leg like a small dog.
Most search and relationship software does not have a feedback mechanism to modify future results. There is no way to slap the software with a rolled-up New York Times edition, yelling, "Bad software! Bad software!" I bet with 30 degrees of seperation you could link George Bush in 3rd grade as a childhood buddy of Osama Bin Laden, Bill Clinton, and Mark Russell (the comedian).
Thanks for the very stimulating article. Two comments:
1. The Tacit software does sound interesting to me, but of course such techniques must be handled with care and sensitivity. The CEO's account reminds me of the far less "sophisticated" (but possibly more effective) way we had of getting similar results at DEC back in the 1980s. Using VAXNotes - roughly analogous to Lotus Notes but much more lightweight and less intrusive - anyone could set up a globally-visible discussion forum on any topic. People interested in that topic naturally gravitated to the relevant Notes file, which over time became a treasure-trove of little-known but important facts. Also, you could ask questions and get answers very quickly - sometimes within minutes. Usenet has some of the strengths of VAXNotes, but also some serious weaknesses. There is no systematic provision for maintaining, indexing and generally valuing "back numbers"; and the signal-to-noise ratio is ridiculous, mainly because Usenet is open to anyone.
2. Thanks for making the Kevin Bacon point - high time someone in the media noticed that. For an encore, do you think you could perhaps start up a campaign to laugh heartily whenever the authorities tell us someone is "linked to" someone or something else? For instance, you are linked to the Madrid bombing - you mentioned it in your article. The beauty of this artfully sloppy phrase is that it can mean anything you want it to. Above all, listeners will tend to assume a more meaningful "linkage" than the speaker would admit to if pressed. Newspeak is not dominant, YET, but there are signs that it is starting up.
The FBI user should have looked further into the results. They couldn't see how the answer was reached, so they should have investigated further. The computer gave them a matched finger print and whatever else it could find out about Brandon Mayfield, so the feds should have sat down and examined the evidence - the search had been narrowed down from X million people to one person - it's not like a slightly more in depth look would have taken months. Simon Cooper
I much enjoyed your piece on this. It should be required reading for all would-be science correspondents, not to mention politicos.
Perhaps you should includes Bill Gates in the circulation, too: "..we've become numbed into thinking that technology always improves" could have been written with a particular operating system in mind...
These Total Obfuscation Unawareness people look like stamp collectors of the worst kind. They think that if they collect as many stamps as possible their collection will become valuable. Add to that their hope that they will eventually find a way to combine many stamps in a "special way" to make a revealing portrait of Osama and uncover all his secrets and you've got a cross between conspiracy paranoids and photo collage hobbyists. These people seem to not use intelligence (of the mental kind) for some reason. Is because it's too expensive? Is it because they don't have it any more? Is it because it can't function without the other kind of intelligence, which is itself lacking? Or is it because they want to obfuscate the way conclusions are reached (perhaps because of some of the previous reasons)? Using a "Holy Grail Tech" rationalization over expert advice *does* fool a lot of people, though hopefully temporarily.
Machines keep lots of information organized, but only humans can evaluate, organize and explore information in a humanly sensible way.®
Sponsored: Customer Identity and Access Management