1. What Do Experts Need to Know? (and When Do they Know Too Much?)
William C. Thompson
University of California, Irvine
Isaac Newton Institute, September 29, 2016

2. What is relevant for an expert?
For trier-of-fact (decision maker)
Evidence is relevant if:
(a) it has any tendency to make a fact more or less probable than it would be without the evidence; and
(b) the fact is of consequence in determining the action.
(Federal Rule of Evidence 401)
Should there be different rules of relevance for experts who are not decision makers?
If so, what should those rules be?

3. Decision structure and information flow—Clinical Medicine
Patient
Primary Physician
Radiologist
Neurologist
Lab Tech

4. Decision structure in biological agent forensics
DOJ
FDA labs
FBI Laboratory
USDA labs
FBI CBSU
CDC labs
NBFAC
National Biodefense Analysis and Countermeasures Center
Chemical Biological Sciences Unit
Commercial labs
Academic labs
National Labs

5. Decision structure and information flow—Judicial System
Judge
Jury
Lay witness
Expert 2
Expert 1
See, Thompson, 2016

6. Concerns About Contextual Bias
NAS Report on Forensic Science (2009):
“Forensic science experts are vulnerable to cognitive and contextual bias…”
“…renders experts vulnerable to making erroneous identifications”
“These disciplines need to develop rigorous protocols to guide these subjective interpretations…”
“…to date there is no good evidence that the forensic science community has made a sufficient effort to address the bias issue…”

7. Contextual bias in forensic science
When a forensic scientist’s judgments or interpretations
are influenced inappropriately
by something irrelevant to their scientific conclusion
i.e., information that is domain-irrelevant or task-irrelevant
But, is there a principled way to determine what is relevant?

8. Taylor et al. (2014) Blood Pattern Analysis Study
When witness’ statements mentioned a gunshot, pattern was more likely to be interpreted as “high velocity” splatter
When witness’ statements mentioned coughing, pattern was more likely to be interpreted as “expiration”
These effects were stronger when
Less information was available (smaller sample)
Pattern more ambiguous due to background

10. Studies Showing Contextual Bias in Forensic Science
Document examination (Miller, 1984)
Fingerprint interpretation (Dror, Charlton & Peron, 2006; Dror & Rosenthal, 2008)
Crime scene analysis (Helsloot and Groenendaal, 2011)
Bite mark analysis (Osborne, Woods, Kieser & Zajac, 2013)
DNA Interpretation (Dror & Hampikian, 2011)
Blood spatter analysis (Taylor et al. 2014)
Forensic Anthropology (Nakhaeizadeh, Dror & Morgan, 2014)

11. Contextual Management Procedures
Case Manager—screens out task-irrelevant information
Sequential Unmasking—delays exposure to relevant information that might be biasing if revealed too early
Evidence may be relevant to some tasks but not others

12. Argument against blinding
For an expert, more information is always better than less
Would you want you doctor to be blind?
Would you put blinders on Sherlock Holmes?
How can it be a bias if it brings me closer to the truth?

13. What is the expert’s role in the justice system?

15. Should a bitemark expert consider DNA evidence?
If I then found that DNA [evidence] came back as not excluding that same person, my confidence level would increase. I might be willing to upgrade my opinion from cannot exclude to probable….Now, many odontologists say you shouldn’t have any awareness of the DNA results compared to the bite mark…but if I subsequently get them, then I reserve the right to write a revised opinion. And I have done that.
(Testimony in NY v. Dean, 2012, RT p. 87)
Is the DNA result “relevant”?
HOW is the DNA relevant?

16. Backward Reasoning
The DNA matches the defendant Defendant is the perpetrator Defendant is probably the source of the bitemark

17. Potential for Double Counting when assessing common source
Expert #1
Expert #2
Examines the samples
Considers likelihood of the observed similarities and differences
if there is a common source
if there is not
Draw a conclusion accordingly
Logic—bottom-up; from Evidence to Conclusion
Scientific
Considers investigative facts of the case to judge whether the samples are likely to have a common source
Draws conclusion accordingly
Logic—top-down (backwards); from conclusion to evidence
Unscientific
Fraudulent (if presented as a scientific opinion)

18. Assessing Whether Samples Have a Common Source
Expert #3 (Hybrid Approach)
Examines evidence like Expert #1
But relies on other evidence (like Expert #2) to resolve uncertainty and determine how strongly to state conclusions
Integrates bottom up and top-down (backward) reasoning
Partly scientific, partly not
Therefore,
Less valuable to the trier-of-fact than conclusion of Expert #1??

19. Maximizing Incremental Probative Value of Expert Evidence
Defendant is the perpetrator?
Defendant is the source of a latent print found at crime scene?
Latent print and defendants print have similar ridge patterns.
What does expert need to know to maximize probative value of R for distinguishing H,NH?

20. In which model is the incremental probative value of R higher?

21. Modeling Contextual Bias
Expert Blind to W
Expert Exposed to W
p(R|H)
p(R|NH)
0.8
0.0001
0.9
0.0003
1.0
0.001 LR
8000
4500
3333
1000

22. The Criminalist’s Paradox
Contextual bias may decrease the probative value of the expert’s conclusion
while also increasing the probability that the expert is correct
By helping themselves be “right,” analysts may increase chances the justice system will go wrong.
See Thompson, Aust.J.Forensic Sci (2011)

23. Forensic scientists should rely solely on task-relevant information when performing forensic analyses
OSAC should specify what types of information are task-relevant for common forensic tasks
Forensic labs should take steps to avoid exposing analysts to task-irrelevant information through the use of context management procedures

24. What Information Is Task-Relevant? Depends on Proposition Addressed
Source
Is S the source of this blood stain?
What caused this blood spatter pattern?
Was homicide the manner of death?
Activity
Most forensic scientists confine themselves to opining on source level propositions. The issue of whether a crime occurred, and what crime it was, is a matter for the legal system (judge or jury) rather than a forensic scientist. An exception is the medical examiner who is sometimes asked to make an independent determination of both cause and manner of death.
Crime
See, Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A hierarchy of propositions: deciding which level to address in casework. Science and Justice 1998; 38:

25. Information is task-relevant if it is necessary for drawing conclusions:
 (i) about the propositions in question,
(ii) from the physical evidence that has been designated for examination,
(iii) through the correct application of an accepted analytic method by a qualified analyst.

26. Examples of Task-Irrelevant Information for Latent Print Analyst
Information about the suspect’s criminal history
Information that suspect confessed
Information that suspect was implicated by other physical evidence at crime scene
Information that another latent print examiner identified the suspect as the source of a print found on a different item at the same crime scene

27. Special Problems of Medical Examiners
Iowa v. Tyler (Iowa Supreme Court, June 30, 2015)
Medical examiner should not have been allowed to express an opinion on the manner of death that was based on non-medical evidence
But ME can rely on non-medical evidence in reaching conclusion for death certificate
Dilemma might be resolved through a blind verification process that focuses on what conclusions, if any, can be drawn from medical evidence alone.

28. Do you have rules or procedures in your laboratory that limit what examiners can know or find out about a case when performing bench-level examinations of specific items of evidence (e.g., latent print or DNA comparisons)?
Answer
Response %
Yes 14
10% No
129
90%
Total
143
100%

29. In your laboratory, do examiners have access to police reports and other background information about the cases when performing bench-level examinations of specific items of evidence (e.g., fingerprint or DNA comparisons)?
Answer
Response %
Yes, all have access to such materials 86
60%
Some have access and some do not 43
30%
No, none have access to such materials 14
10%
Total
143
100%

30. Do you allow examiners who are performing bench-level examinations of specific items of evidence to communicate directly with police officers and other officials about submitted evidence prior to the conclusion of the examination and analysis?
Answer
Response %
Yes, all are allowed
126
88%
Some are allowed and some are not 12 8%
No, none are allowed 6 4%
Total
144
100%

31. Emotional involvement with cases
DNA Lab Notes
“I asked how they got their suspect. He is a convicted rapist and the MO matches the former rape…The suspect was recently released from prison and works in the same building as the victim…She was afraid of him. Also his demeanor was suspicious when they brought him in for questioning…He also fits the general description of the man witnesses saw leaving the area on the night they think she died…So, I said, you basically have nothing to connect him directly with the murder (unless we find his DNA). He said yes.”

32. Being Scientifically Rigorous or Being Sherlock?
Forensic scientists can do both!
Just not at the same time.

33. Argument Against Blinding
We’re not biased (and we’re insulted that you have suggested we are)!!
“I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort. If we are unable to acknowledge and compensate for bias, we have no business in our profession to begin with, and certainly no legitimate plea to the indulgence of the legal system.”
John Thornton, J.Forensic Sciences, 2010).

34. Response to Thornton
“Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity.”
(Thompson et al. Response to Thornton, JFS 2010)

35. Thank you

36. The marvel is not that the bear dances well, but that it dances at all.
Russian proverb