How do we assess the expert we consult, the one we hire and proffer, or the expert we will be confronting?  If it is the opposing expert there will be a deposition (in a civil case) and testimony from prior cases as well as intelligence gathering from attorneys in your field who have faced the same witness.  If it is your expert it is prior experience, the recommendation of colleagues, and possibly a matter of cost and availability.

Regardless of which side the expert is on, it is the curriculum vitae that will be key in the initially evaluating the witness’ bona fides.  But CVs are more than skin deep – and three issues may be lurking beneath the surface that require assessment.

The first is the witness’ list of publications.  Multiple concerns arise here.  A claim that a publication is “peer reviewed” is meaningless unless assessed by two criteria – who are the peers and is the review “ double blind?”  The criticism leveled in a recent court decision regarding firearm experts explains the concerns:

The…peer review process itself is “open,” meaning that both the author and reviewer know the other’s identity and may contact each other during the review process…This open process seems highly unusual for the publication of empirical scientific research… The practice of double-blind peer review, by contrast, constitutes the standard among scientific publications and guards against personal and institutional biases by shielding both reviewer and author from the identity of the other…Use of a so-called “open” process diminishes the extent to which proponents of firearms and toolmark identification evidence can claim that its articles have been subjected to meaningful, stringent peer review.

…[U]nlike other scientific journals, the AFTE Journal is not more broadly available and cannot even be obtained in university libraries. Id. 18:11-13. Such restricted access effectively forecloses the type of review of the journal’s publications by a wider community of scientists, academics, and other interested parties that could serve as an important mechanism for quality assurance.

…

[G]enerally limiting the review of its publications and making them available only to its members or others who pay[] avoids the scrutiny of scientists and academics outside the field of firearms and toolmark analysis. Third, the very nature of AFTE impacts the meaningfulness of …[T]hose who review the…articles may be trained and experienced in the field of firearms and toolmark examination, but do not necessarily have any specialized or even relevant training in research design and methodology. Perhaps more importantly, members of the…editorial board—those who review its articles prior to publication—have a vested, career-based interest in publishing studies that validate their own field and methodologies.

United States v. Tibbs, 2019 D.C. Super. LEXIS 9, *29-35.

Separate from peer review is the quality of the journal itself.  Professor David Kaye has developed a blog tracking questionable scientific journals – in one post he dissects the invitation to publish offered by one with an impressive name:

JScholar‘s Journal of Forensic Research and Crime Studies (JFRCS) promises “maintaining rigorous peer-review process.” A competent academic journal would use referees to suggest improvements in most articles they recommend for acceptance, but the JFRCS reviewers need not spend much time on that because “All accepted papers will be appeared online immediately” and online is the only mode of publication. Instant publication also helps lighten the editors’ workload.

Kaye notes an additional attribute: “The cost for the privilege of having work posted in this journal is $777.”  http://flakyj.blogspot.com/ (last visited November 30, 2019).  So the lessons for a list of “impressive” publications is to read on – assess who the reviewers are, whether they are blinded, and if you the attorney can also pay for the privilege of publishing in that same journal.

Pay for play applies not only to journals but to organization membership and certification.  The infamous story is that of the American College of Forensic Examiners Institute, which permits one to receive a certificate as a Certified Forensic Consultant upon payment of a fee and passing an online multiple choice exam.  Among those with this impressive credential is a house cat, registered by its owner.  https://www.pbs.org/wgbh/frontline/article/no-forensic-background-no-problem/ (last visited November 30, 2019).  The lesson here – see how easy it is for you, the non-scientist or non-engineer lawyer – to become a member of the organization and thereafter be credentialed.

The third consideration in weighing the expert’s strengths involves “proficiency tests.”  In many fields, analysts are required to take periodic tests to assess “proficiency” – the ability to perform tasks in the discipline.  The problem here is simple – sometimes the stringency [difficulty level] of those tests is quite low, resulting in easy success and perfect scores.

In the just-published report “Fingerprint Analysis Is High-Stakes Work — But It Doesn’t Take Much To Qualify As An Expert,” the INTERCEPT reports of three public defendens who paid for and took the proficiency test taken by latent print examiners.  https://theintercept.com/2019/11/29/fingerprint-examination-proficiency-test-forensic-science/  Each had a near-perfect score,  11 out of 12.  “Each of them had correctly identified all but one of the fingerprints contained in the test.”  The lesson here?  A proficiency test is only as meaningful as its level of difficulty – but that can’t be ascertained without taking it yourself or finding reports confirming how easy the particular test is.

We need and value experts — but lawyers vetting those witnesses need to become expert in reading between and behind the lines of a C.V.