Steve Lund and Hari Iyer steve.lund@nist.gov hari@nist.gov Considering the Reliability of and Alternatives to Likelihood Ratios Used to Explain Forensic Evidence Steve Lund and Hari Iyer steve.lund@nist.gov hari@nist.gov

Disclaimer Viewpoints expressed are our own and are not intended to reflect the viewpoints of anyone else at NIST We are only discussing what makes sense to us and are not trying to claim that these are new perspectives

Illustration: Comparing Toolmark Striae Q

Comparing Toolmark Striae – Illustration Q After digitally capturing and computer aligning of the striae from R and Q we can determine the maximum number of consecutively matching striae.

Comparing Toolmark Striae – Specific Case Q The maximum number of consecutive matching striae is 6

Reference Collection Ground truth known 4

Reference Collection Ground truth known

4 Reference Collection Ground truth known Chance of getting a 4 when comparing marks from the same tool: 0.0159 0.0262 0.0471 2.52e-07 0.0006 0.0060 Chance of getting a 4 when comparing marks from different tools: Min = 2.65

4 Reference Collection Ground truth known Chance of getting a 4 when comparing marks from the same tool: 0.0159 0.0262 0.0471 2.52e-07 0.0006 0.0060 Chance of getting a 4 when comparing marks from different tools: Min = 2.65; Max=187000

Some Comments on Probability Probability is not a description of what’s been observed Some might say it’s a personal belief, based on what information is currently available Others might view it as a best guess for what would be observed, if we could collect unlimited data Under either case, there is no single right answer.

Some Comments on LRs Under the perspective of personal probabilities, the question is what is my LR, not what is the LR of any one particular expert Expert can’t create an exact answer for anyone else’s LR, but may be able to characterize a range under different assumptions Asking multiple experts may illustrate spread (at least partially) Under the perspective of predicting what unlimited data would illuminate, use statistical inference tools to assess whether each considered model is consistent or inconsistent with currently available data LR value is plausible if it corresponds to at least one plausible model LR value only implausible if there’s no corresponding plausible model – this can be hard to demonstrate

Some Comments on LRs What’s the best value? Under either perspective of probability a best guess is expected to be wrong an uncertainty is intended to characterize how far off you might be What’s the best value? Is the offered value reasonable? What is the range of “reasonable” interpretations?

Reliability Our working notion: To be reliable, a report/testimony must have a low risk of being wrong (in a way that affects the ultimate decision)

Reliability: Expert’s intended meaning

Reliability: Recipient’s interpretation Rule 403. Excluding Relevant Evidence for Prejudice, Confusion, Waste of Time, or Other Reasons The court may exclude relevant evidence if its probative value is substantially outweighed by a danger of one or more of the following: unfair prejudice, confusing the issues, misleading the jury, undue delay, wasting time, or needlessly presenting cumulative evidence.

LR Reliability Concern Don’t want experts or recipients to falsely believe that any reasonable LR assessment would differ only negligibly from offered value Guard against extreme values resulting from choice of model

LR Reliability Concern Don’t want recipients to falsely believe that any reasonable LR assessment would differ only negligibly from offered value Guard against extreme values resulting from choice of model If providing an LR, try to investigate and express the range of reasonable values

Among different forms for expressing expert opinion (e. g Among different forms for expressing expert opinion (e.g., categorical conclusions or probabilities of source), LRs have some clear advantages Federal Rules of Evidence permit but do not advocate for experts communicating in the form of an opinion Experts could focus on presenting an overview of the information available to them that would serve as the basis of an expert interpretation (cards are on the table)

Reference Collection Ground truth known 11

Thank you!