What Do Experts Need to Know? (and When Do they Know Too Much?) William C. Thompson University of California, Irvine Isaac Newton Institute, September 29, 2016

What is relevant for an expert? For trier-of-fact (decision maker) Evidence is relevant if: (a) it has any tendency to make a fact more or less probable than it would be without the evidence; and (b) the fact is of consequence in determining the action. (Federal Rule of Evidence 401) ------------------------------------------------------------------------- Should there be different rules of relevance for experts who are not decision makers? If so, what should those rules be?

Decision structure and information flow—Clinical Medicine Patient Primary Physician Radiologist Neurologist Lab Tech

Decision structure in biological agent forensics DOJ FDA labs FBI Laboratory USDA labs FBI CBSU CDC labs NBFAC National Biodefense Analysis and Countermeasures Center Chemical Biological Sciences Unit Commercial labs Academic labs National Labs

Decision structure and information flow—Judicial System Judge Jury Lay witness Expert 2 Expert 1 See, Thompson, 2016

Concerns About Contextual Bias NAS Report on Forensic Science (2009): “Forensic science experts are vulnerable to cognitive and contextual bias…” “…renders experts vulnerable to making erroneous identifications” “These disciplines need to develop rigorous protocols to guide these subjective interpretations…” “…to date there is no good evidence that the forensic science community has made a sufficient effort to address the bias issue…”

Contextual bias in forensic science When a forensic scientist’s judgments or interpretations are influenced inappropriately by something irrelevant to their scientific conclusion i.e., information that is domain-irrelevant or task-irrelevant But, is there a principled way to determine what is relevant?

Taylor et al. (2014) Blood Pattern Analysis Study When witness’ statements mentioned a gunshot, pattern was more likely to be interpreted as “high velocity” splatter When witness’ statements mentioned coughing, pattern was more likely to be interpreted as “expiration” These effects were stronger when Less information was available (smaller sample) Pattern more ambiguous due to background

Studies Showing Contextual Bias in Forensic Science Document examination (Miller, 1984) Fingerprint interpretation (Dror, Charlton & Peron, 2006; Dror & Rosenthal, 2008) Crime scene analysis (Helsloot and Groenendaal, 2011) Bite mark analysis (Osborne, Woods, Kieser & Zajac, 2013) DNA Interpretation (Dror & Hampikian, 2011) Blood spatter analysis (Taylor et al. 2014) Forensic Anthropology (Nakhaeizadeh, Dror & Morgan, 2014)

Contextual Management Procedures Case Manager—screens out task-irrelevant information Sequential Unmasking—delays exposure to relevant information that might be biasing if revealed too early Evidence may be relevant to some tasks but not others

Argument against blinding For an expert, more information is always better than less Would you want you doctor to be blind? Would you put blinders on Sherlock Holmes? How can it be a bias if it brings me closer to the truth?

What is the expert’s role in the justice system?

Should a bitemark expert consider DNA evidence? If I then found that DNA [evidence] came back as not excluding that same person, my confidence level would increase. I might be willing to upgrade my opinion from cannot exclude to probable….Now, many odontologists say you shouldn’t have any awareness of the DNA results compared to the bite mark…but if I subsequently get them, then I reserve the right to write a revised opinion. And I have done that. (Testimony in NY v. Dean, 2012, RT p. 87) Is the DNA result “relevant”? HOW is the DNA relevant?

Backward Reasoning The DNA matches the defendant Defendant is the perpetrator Defendant is probably the source of the bitemark

Potential for Double Counting when assessing common source Expert #1 Expert #2 Examines the samples Considers likelihood of the observed similarities and differences if there is a common source if there is not Draw a conclusion accordingly Logic—bottom-up; from Evidence to Conclusion Scientific Considers investigative facts of the case to judge whether the samples are likely to have a common source Draws conclusion accordingly Logic—top-down (backwards); from conclusion to evidence Unscientific Fraudulent (if presented as a scientific opinion)

Assessing Whether Samples Have a Common Source Expert #3 (Hybrid Approach) Examines evidence like Expert #1 But relies on other evidence (like Expert #2) to resolve uncertainty and determine how strongly to state conclusions Integrates bottom up and top-down (backward) reasoning Partly scientific, partly not Therefore, Less valuable to the trier-of-fact than conclusion of Expert #1??

Maximizing Incremental Probative Value of Expert Evidence Defendant is the perpetrator? Defendant is the source of a latent print found at crime scene? Latent print and defendants print have similar ridge patterns. What does expert need to know to maximize probative value of R for distinguishing H,NH?

In which model is the incremental probative value of R higher?

Modeling Contextual Bias Expert Blind to W Expert Exposed to W p(R|H) p(R|NH) 0.8 0.0001 0.9 0.0003 1.0 0.001 LR 8000 4500 3333 1000

The Criminalist’s Paradox Contextual bias may decrease the probative value of the expert’s conclusion while also increasing the probability that the expert is correct By helping themselves be “right,” analysts may increase chances the justice system will go wrong. See Thompson, Aust.J.Forensic Sci (2011)

Forensic scientists should rely solely on task-relevant information when performing forensic analyses OSAC should specify what types of information are task-relevant for common forensic tasks Forensic labs should take steps to avoid exposing analysts to task-irrelevant information through the use of context management procedures

What Information Is Task-Relevant? Depends on Proposition Addressed Source Is S the source of this blood stain? What caused this blood spatter pattern? Was homicide the manner of death? Activity Most forensic scientists confine themselves to opining on source level propositions. The issue of whether a crime occurred, and what crime it was, is a matter for the legal system (judge or jury) rather than a forensic scientist. An exception is the medical examiner who is sometimes asked to make an independent determination of both cause and manner of death. Crime See, Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A hierarchy of propositions: deciding which level to address in casework. Science and Justice 1998; 38: 231-239.

Information is task-relevant if it is necessary for drawing conclusions:  (i) about the propositions in question, (ii) from the physical evidence that has been designated for examination, (iii) through the correct application of an accepted analytic method by a qualified analyst.

Examples of Task-Irrelevant Information for Latent Print Analyst Information about the suspect’s criminal history Information that suspect confessed Information that suspect was implicated by other physical evidence at crime scene Information that another latent print examiner identified the suspect as the source of a print found on a different item at the same crime scene

Special Problems of Medical Examiners Iowa v. Tyler (Iowa Supreme Court, June 30, 2015) Medical examiner should not have been allowed to express an opinion on the manner of death that was based on non-medical evidence But ME can rely on non-medical evidence in reaching conclusion for death certificate Dilemma might be resolved through a blind verification process that focuses on what conclusions, if any, can be drawn from medical evidence alone.

Do you have rules or procedures in your laboratory that limit what examiners can know or find out about a case when performing bench-level examinations of specific items of evidence (e.g., latent print or DNA comparisons)? Answer Response % Yes 14 10% No 129 90% Total 143 100%

In your laboratory, do examiners have access to police reports and other background information about the cases when performing bench-level examinations of specific items of evidence (e.g., fingerprint or DNA comparisons)? Answer Response % Yes, all have access to such materials 86 60% Some have access and some do not 43 30% No, none have access to such materials 14 10% Total 143 100%

Do you allow examiners who are performing bench-level examinations of specific items of evidence to communicate directly with police officers and other officials about submitted evidence prior to the conclusion of the examination and analysis? Answer Response % Yes, all are allowed 126 88% Some are allowed and some are not 12 8% No, none are allowed 6 4% Total 144 100%

Emotional involvement with cases DNA Lab Notes “I asked how they got their suspect. He is a convicted rapist and the MO matches the former rape…The suspect was recently released from prison and works in the same building as the victim…She was afraid of him. Also his demeanor was suspicious when they brought him in for questioning…He also fits the general description of the man witnesses saw leaving the area on the night they think she died…So, I said, you basically have nothing to connect him directly with the murder (unless we find his DNA). He said yes.”

Being Scientifically Rigorous or Being Sherlock? Forensic scientists can do both! Just not at the same time.

Argument Against Blinding We’re not biased (and we’re insulted that you have suggested we are)!! “I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort. If we are unable to acknowledge and compensate for bias, we have no business in our profession to begin with, and certainly no legitimate plea to the indulgence of the legal system.” John Thornton, J.Forensic Sciences, 2010).

Response to Thornton “Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity.” (Thompson et al. Response to Thornton, JFS 2010)

Thank you Email: william.thompson@uci.edu

The marvel is not that the bear dances well, but that it dances at all. Russian proverb