The Realm of Junk Science
Search the internet, and you can find links to articles and discussions of ‘lip print’ and ‘ear print’ comparison as forensic tools for identifying perpetrators. Or search U.S. decisional law, and you may find a transcript of testimony like the following:
Q:…[D]o you have an opinion as to the probability that the defendant’s left ear is the source of the latent impression which was left at the scene of the crime in this case?
A: I do have an opinion, yes.
Q: What is your opinion, then?
A: I think it’s probable that it’s the defendant’s ear is the one that was found on the scene.
. . . .
Q: [H]ow confident are you of the opinion that you just expressed?
A: I’m 100 percent confident with that opinion
State v. Kunze, 97 Wn. App. 832, 848-849 (Wash.Ct.App. 1999). The appellate court repudiated such testimony as not having gained general acceptance; but that a trial court admitted the same with no data base of ear prints, no standards across the field for how an ear print analysis is to be conducted, and no error rate evidence, should give pause.
That example of ‘junk science,’ the term popularized by Peter Huber 30-plus years ago, is from roughly a quarter-century ago and thus may be dismissed as an anomaly, a vestige of a less scientific, less skeptical time. But the new book JUNK SCIENCE by Chris Fabricant, the director of strategic litigation for the Innocence Project, forcefully reminds us that in the arena of forensic discipline testimony an acceptance of ‘junk’ information prevails. With no data-base for comparisons, no universal standards within a discipline, no inter-evaluator consistency in outcomes – essentially, no “there, there” – people have been imprisoned on questionable claims of bitemark identification and individualization.
JUNK SCIENCE is beautifully written, a compilation of stories of trials, appeals, the vagaries of purported science in the courtroom and the author’s personal journey across two landscapes – the many states where he went to challenge bitemark or similar evidence and his own journey of growth as a lawyer and friend to those jailed based on what is at times head-shakingly bad proof. JUNK SCIENCE also details how a single discipline can self-create a veneer of professionalism and accreditation. Among many other legal issues, it raises the question of who the relevant scientific community is – the narrow set of practitioners of a particular discipline, or the broader one of scientists, statisticians, and those who teach/practice in the broader realm of forensic science. While courts have deferred to the former, JUNK SCIENCE compellingly shows the latter to be the proper source for determining reliability and trustworthiness.
Put simply, JUNK SCIENCE is essential reading for anyone who will present, challenge or judge purported science in the courtroom, and on a broader level for anyone concerned about accuracy in adjudication. Below is an interview with its author.
* * * * *
It is 2022, and to date 375 people in the United States have been exonerated by DNA testing, including 21 who served time on death row. With that awareness, is the fear of junk science in the courtroom outdated?
One thing we have to keep in mind is that the 375 innocent people unquestionably represent the tip of the iceberg. The National Registry of Exonerations lists over 3,000 people, and that number too far understates the likely number of innocent in prison in this country, which at any given time incarcerates about 2 million Americans. If only 1% are factually innocent we’re talking about tens of thousands of people. And, just focusing on the DNA exonerations, unreliable forensic sciences played a role in nearly half of those cases. So the threat that junk science poses in convicting – even executing – the innocent cannot be overstated.
You have litigated challenges to junk science under Daubert and Frye. Are there advantages/disadvantages to each?
I think fair application of either standard would exclude most unreliable evidence, though I tend to favor Daubert. Frye litigation depends on defensible definitions of the “relevant scientific community”. I talk in my book about how courts have really failed here in Frye litigation, since, incredibly, judges have typically defined that community as practitioners of the technique at issue – the same people who make their living testifying as expert witnesses. So, when courts ask these experts whether their science is generally accepted – while excluding mainstream scientists from the inquiry, who have been highly critical – we run into the famous Upton Sinclair quote: “It is difficult to get a man to understand something when his salary depends on his not understanding it.” Also, many jurisdictions apply Frye only to “novel” forensic techniques, which is absurd. I have, many times, had to contend with arguments that junk science – bite mark evidence, for example – cannot be challenged because it’s not novel. This is of course totally contradictory to science, which is a process always moving forward, always abandoning previously held beliefs when research falsifies previously held beliefs. If courts defined the relevant community to include mainstream scientists, people with no stake in the outcome of the decision, and factored changes in scientific understanding over time into decision-making, Frye would work as intended. Indeed, the original Frye decision, which involved an early lie detector test, didn’t limit the relevant community to proponents of the new technology.
Faithful application of Daubert would likewise exclude unreliable evidence. The trouble, of course, is that, like Frye, it’s not applied faithfully. If it was, how would so many techniques, such as firearms analysis, be allowed into court as “scientific” evidence without valid error rates? I litigated a case in March in which an expert confidently testified, in 2022, that his error rate was “zero.”
Your book refers repeatedly to forensic discipline evidence as “poor people science.” What is this meant to connote?
“Poor people science” is a theme that I explore throughout Junk Science, beginning with the chapter on the Daubert decision in 1993, which led to the exclusion of a lot of speculative expert testimony in civil litigation, but did nothing to rein in junk science in the criminal legal system. Poor people science is the difference between the scientific evidence used in civil litigation, where money is at stake and less likely to be unreliable, and the “scientific” evidence used in the American criminal justice system, where life and liberty are at stake, and junk science is pervasive. The difference speaks to our values as a society.
Most judges and lawyers are not science-educated or trained. What concept has been most difficult to translate or communicate to non-scientists?
Statistics. The lack of a valid statistical foundation underlies the problems associated with so many forensic techniques, and most lawyers have no understanding of even basic statistics.
The book focuses in large part on two disciplines – fire investigation and ‘bitemark matching.’ What is the greatest flaw of, or concern about, each?
Excessive subjectively, which really is an issue in virtually all forensic sciences. The pernicious influence of cognitive bias has been demonstrated in social science research extensively, including DNA mixture interpretation, forensic pathology, fingerprints, forensic anthropology, dog scent evidence, the list goes on. In mainstream science, this phenomenon is well-understood and measures are taken to mitigate the influence. As I discuss in the chapter on bias, virtually no fields of forensics blind the experts from irrelevant information, e.g., the prosecution theory of guilt, the race of the suspect, the suspect’s criminal record, details about the victim – all of which are routinely included in chain of custody documents and provided to experts in advance of their analysis.
If there were one reform you would like to see enacted, what would it be?
Researchers in collaboration with lab practitioners – not attorneys, law enforcement – should lead and guide the work on forensic science research and standard-setting. The federal government should make significant investments in both basic and applied research related to forensic science, led by scientists and the federal scientific entities such as the National Science Foundation (NSF) and the National Institute of Standards and Technology (NIST). Additionally, the federal government should develop a comprehensive federal research agenda, again led by scientists and federal scientific entities, so that there is a clear understanding of what research already exists and what is most urgently needed, and a coordinated effort among all agencies involved with forensic science-related research in any way.
What is on the horizon in addressing forensic discipline testimony and proof?
What really concerns me is the increasing use of proprietary software programs in criminal trials. Most of this technology – such as probabilistic genotyping software programs, facial recognition, ShotSpotter technology, which I believe to be highly unreliable – are essentially black boxes. The companies that produce these products are in business to make money, not necessarily to advance justice. When defense attorneys have sought access to internal source codes to examine the reliability of the evidence being proffered against their clients, the companies have fought tooth and nail to prevent disclosure, even under protective orders. Unbelievably, and consistent with the poor people science theme in Junk Science, corporate America has been winning these arguments; many courts have elevated “trade secrets” above the fundamental 6th Amendment right to confront “witnesses” against you, and in this case that witness is the source code.
* * * * *
The outcry against junk science is not new, at least in personal injury and similar contexts. Myers, “The Battle of the Experts:” A New Approach to an Old Problem in Medical Testimony, 44 NEB. L. REV. 539, 543 (1965) (abuse of expert testimony constitutes “‘a painful canker in the body of the law of evidence which has been festering for many decades’”). As one federal judge bemoaned regarding a particular pair of experts, “[t]his Court is disappointed with the apparent fact that these so-called experts can take such license from the witness stand; these witnesses say and conclude things which, in the Court’s view, they would not dare report in a peer-reviewed format.” Johnston v. United States, 597 F. Supp. 374, 415 (D. Kan. 1984). Yet such skepticism has been absent in criminal cases all too often.
Bitemark evidence continues to be proffered to this day. What JUNK SCIENCE reminds us all of is that science needs to be fully vetted, often a task that a criminal trial is ill suited for; and until it is, the use of questionable proof will undoubtedly lead to wrongful convictions (and with them, the failure to pursue and identify the actual perpetrator).