Statistics and the Law Varieties of Statistical Challenges Varieties of Challenges to Statistics

Conforming Statistics to the Law The law sets parameters of relevance and poses the questions the statistician must answer. – May not be questions the statistical expert is best positioned to answer. – Standards of proof may not be statistical standards.

Legal and Statistical Models: Best Case Jury Discrimination: – Statistical and legal issues almost perfectly aligned. Jury selection is supposed to be random. Easy to model the extent to which a given distribution is consistent with a random draw from a given population. – Challenges: Determining the characteristics of the relevant population. Determining how large a deviation is required to trigger a legal finding.

Castenada v. Partida, 430 U.S. 432 (1977) “As a general rule for such large samples, if the difference between the expected value and the observed number is greater than two or three standard deviation, then the hypothesis that the jury drawing was random would be suspect to a social scientist.” Grand Jury case, not a petit jury – “Key Man” system was permitted. – Mexican-Americans in county dominated politically by Mexican (79% of county; 50% of grand jury) Married women with changed names. Non-citizen immigrants.

Legal and Statistical Models: A Not So Good Case Regression Models – Employment discrimination, antitrust, etc. Challenges – The right legal model Sobel v. Yeshiva University, 839 F2d 18 (1988) – The right scientific model Difficult to determine – Functional Form (OLS is default) – Included Variables » Whose burden? Smith v. VCU, 84 F.3d 672 (1995)

Model Mischief Making Challenge: Ensure fair, transparent, relevant and honest statistics. The law does not make this easy. – Choosing models by their outcomes. Variable choices – E.g. minimizing effects by adding proxies – P Hacking E.g. Concealing approaches tried. – Avoiding robustness checks Love the results – top there – Ignoring effect sizes Increase sample size – get significance. Pretend that only significance matters. – Unjustified assumptions

Answering the Law’s Question DNA evidence. – Science question: how likely is it that the DNA of a “random person” will, like the defendant’s DNA, match the crime scene sample. In usual case, easy to answer with conservative adjustments. – Law’s question: how likely is testimony reporting a DNA match if the defendant did not leave the crime scene sample. This requires accounting for: Random match probability given suspect population Likelihood of laboratory error – Sample contamination – Mislabeling – Subjective judgments – Perjury Complication:.001 probability of receiving testimony of DNA match not the same as.001 probability that a random person would have matching DNA. – Latter implies other matching people. (There is something to the “Defense Attorney’s Fallacy.”)

Other Identification Evidence Similar analysis applies to other evidence that seeks to associate individuals with trace evidence. (e.g. Fingerprints, Handwriting, Bite Marks, Shoe or Tire Marks, Tool Marks, Etc.) – Objective other match probabilities higher. – Uncertainty greater. – Subjectivity greater.

Acquiring Necessary Statistics (Data) Forensic evidence in general – Need to estimate error rates (even for most reliable; e.g., DNA, Chemical Analysis) Proxies (lab acredidation, individual certification, training, experience) of uncertain value. Proficiency testing. – Open: at best sets lower error bound. – Blind: expensive and hard to do. History of error. – Claimed or real correction. Perjury almost impossible to estimate.

Data Needs Many forensic technologies – Need to validate (set bounds on) assumptions. Are fingerprints unique. – How much information is needed for unique identification » Partial prints » Smudged prints – How unique are bite marks? – What do we know about blood splatter. » Arson: A cautionary tale. Need to assess utility. – How much better do handwriting experts do than lay jurors?

Communication Challenges Sweet spot: Fair and appropriate persuasiveness. – Incentives run contrary. Offeror wants to exaggerate probative value. Opponent wants to denigrate probative value. Making statistics understandable. – Difficult to do, especially when both objective match rates and false reporting rates should both be made available. Jay Koehler shoe print study Bayesian dreams.