UNCLASSIFIED Defense Forensic Science Center Evaluating Probabilistic Genotyping Results Joel Sutton, Technical Leader USACIL DNA Casework Branch 7 th Annual Prescriptions for Criminal Justice Forensics, 3 June 2016

UNCLASSIFIED Disclaimer The opinions or assertions contained herein are the private views of the author and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense. Names of commercial manufacturers or products included are incidental only, and inclusion does not imply endorsement by the authors, DFSC, OPMG, DA or DoD. Unless otherwise noted, all figures, diagrams, media, and other materials used in this presentation are created by the respective author(s) and contributor(s) of the presentation and research. 2

UNCLASSIFIED Background Various methods for interpreting/reporting results Testing different LR propositions Testing multiple persons of interest Verbal scale Uninformative – the “new inconclusive” Admissibility and testimony strategies 3 Outline

UNCLASSIFIED The Defense Forensic Science Center has been online using probabilistic genotyping (PG) exclusively in casework since Nov 2014 Our primary casework is sexual assault evidence At the current time, we have testified in 18 court- martials using PG results One thing we have learned is that the validation of the software turned out to be the “easy” part Interpreting and reporting the results has actually been the real challenge It’s not even the “PG” part that is the challenge, but more with the likelihood ratio we are reporting 4 Background

UNCLASSIFIED As laboratories continue to move towards interpreting and reporting DNA results with probabilistic genotyping, there are considerations the legal community needs to be aware of  Training practitioners  Training stakeholders The following slides give examples for some of these considerations 5 Practical Considerations

UNCLASSIFIED DNA results traditionally have one of three conclusions:  inclusion  exclusion  inconclusive PG is based on probabilities, most use a traditional LR framework Some have advocated no longer using terms like “inclusion” or “exclusion” for PG conclusions, but instead simply report out the LR and let the number speak for itself  An LR > 1 favors the prosecution hypothesis  An LR < 1 favors the defense hypothesis  An LR =1 is considered neutral 6 Methods For Interpreting/Reporting Results This is the basic LR formula w/o adding in the additional probabilities for drop

UNCLASSIFIED This has led to two general schools of thought with interpreting PG results based on an LR  Run all mixtures through the program  Evaluate all the data first to determine suitability and try to exclude the POI first 7 Methods For Interpreting/Reporting Results There are pros and cons to each approach

UNCLASSIFIED Methods For Interpreting/Reporting Results Run everything through Pro: considered less subjective Pro: with complex mixtures, can you really ever “include” or “exclude” Con: “junk in, junk out” Con: who’s the expert Interpret first Pro: if one can interpret first, you can “weed out” some of the more difficult mixtures as being uninterpretable Pro: The analyst still serves as the expert for determining inclusions/exclusions Con: There is some subjectivity with this in terms of interpreting complex mixtures and making decisions up front 8

UNCLASSIFIED Hypotheses are also referred to as propositions Assumptions considered for proposition  Number of contributors in the mixture  Any known contributors Depending on these propositions, you can get very different LRs When setting propositions, there are a few things to keep in mind:  We generally know the prosecution hypothesis  We never know nor should we predict what the defense hypothesis will be However, knowing the math behind the LR, we can predict:  What is most reasonable  What is most informative 9 Testing Different Hypotheses

UNCLASSIFIED Sexual Assault case – evidence is semen on a vaginal swab from Victim (V) DNA results - mixture of two individuals, with all the alleles consistent with the V and Suspect (S) Prosecution hypothesis – V + S Defense hypothesis options – unknown individual (U) contributed the semen  V+U  U+U 10 LR Example

UNCLASSIFIED (V+S/U+U) – this LR would be large  The more U on bottom compared to top LR goes “waaay” up  Has to do with the math (V+S/V+U) is smaller  The ratio of U in the LR has been minimized  Reasonable to assume the V on her own swab  Genotype possibilities have been constrained to the foreign DNA contributor which makes the LR lower but also more informative (less genotype possibilities promote better exclusionary power) 11 LR Example

UNCLASSIFIED It is conventional wisdom to try and minimize the ratio of unknowns with the LR where possible This reduces the LR (gives a more conservative estimate) Otherwise the LR can blow up based on the math which does not benefit the accused Making reasonable assumptions for known contributors and keeping the number of contributors the same between propositions is generally preferred If unsure, test and include multiple propositions to assess at trial 12 Which Proposition To Use

UNCLASSIFIED Two choices to make with your proposition:  Evaluate each individual separately in the mixture  Evaluate them together in the mixture If your software can, better to evaluate together Case circumstances and pre-trial conferences will dictate which proposition will be reasonable for the defense 13 Multiple Persons Of Interest

UNCLASSIFIED Often these numbers associated with the LRs being reported will be drastically different What does “50 times more likely” mean compared to “50 septillion times more likely” The differences in the numbers speak to the weight A verbal scale is mostly arbitrary, but may help in verbalizing what these differences mean 14 Verbal Scale

UNCLASSIFIED LR =Verbalized as… 10 billion or greater“Very strong support” 10 million to 10 billion“Strong support” 10,000 to 10 million“Moderate support” 1000 to 10,000“Weak support” >10 to 1000“Very weak support” 1 to 10“Uninformative” 0 to 1“Supports an exclusion” 15 USACIL DNA Verbal Scale Using STRmix™ This scale is data dependent, based on the deconvolution, and used when requested to put the numbers in some sort of general context

UNCLASSIFIED 16 What Does “Uninformative” Mean? This paper demonstrates that LR distributions for competing propositions are very well separated with good quality data, but converge to 1 with low quality data. At that point, the LR becomes uninformative for inclusions/exclusions, essentially inconclusive for either proposition

UNCLASSIFIED 17 Admissibility And Courtroom Testimony This paper drives home some important points on this subject and introduces the concept of a software-expert pair (SEP)

UNCLASSIFIED From the authors’ point of view PG is reliable if properly validated  ISFG recommendations  SWGDAM validation guidelines  OSAC is writing validation standards Peer-reviewed papers from the developers is crucial – the software may be new, but not the concepts The concepts of an LR, MCMC, modeling stutter and peak height variability, and calculating match probabilities are not novel and well-published 18 Admissibility

UNCLASSIFIED The courts expect the results and any software not only be properly validated, but also that it is appropriately applied “Appropriately applied” relates to the expert using it and testifying to it. The software and expert work together (SEP) “It is imperative to note that to have a successful SEP in use for forensic casework requires not only a validated software tool, but a fully trained and competent expert that uses the chosen software” 19 Courtroom Testimony

UNCLASSIFIED FRE 702 speaks to experts and the right for the defense to confront these witnesses It’s the expert after all, not the software that will have to testify “It is accepted that no analyst is required to understand the mathematics and computer program to the extent that they could recreate the system, except the developers themselves. However it is an expectation that analysts at least understand the workings of any system they use to be able to understand and explain the results.” 20 Admissibility And Courtroom Testimony

UNCLASSIFIED Summary Lots of PG systems out there, all slightly different Individual laboratory protocols may use the same system differently Read the literature to better understand the fundamental concepts – training staff is key Understand the LR framework These are not expert systems – do not treat it as a “black box” Work with your customers and stakeholders – they need to be trained as well 21

UNCLASSIFIED Joel Sutton USACIL DNA Technical Leader Questions? Thank you for your time!