1. Black Boxes and Due Process: Transparency in Expert Software Systems AAFS 2016 By Nathan Adams – Adams@Bioforensics.com Dan Krane – Dan.Krane@Wright.eduAdams@Bioforensics.comDan.Krane@Wright.edu

2. Financial disclosure Employees of Forensic Bioinformatic Services, Inc.

3. Laboratory processes What to do How to do it Do it Did it? Done Developmental validation Internal validation Laboratory protocols Casework Review Report Casework

4. “Black box” “A device which performs intricate functions but whose internal mechanism may not readily be inspected or understood; (hence) any component of a system specified only in terms of the relationship between inputs and outputs.” "black box, n." OED Online. Oxford University Press, December 2015. Web. 19 February 2016.

5. Validating likelihood ratios Physical measuring can be validated (underlying true value) Likelihood ratios (LRs) are more abstract, but we can: Study underlying models Measure with “performance tests” Steele, Christopher D., and David J. Balding. "Statistical evaluation of forensic DNA profile evidence." Annual Review of Statistics and Its Application 1 (2014): 361-384.

6. What can we do? Validation studies “The system, as a whole, generates reasonable answers.” Systemic, holistic, sum of the parts Examine the underlying models “These models are independently reliable.” Each part works right

7. What can we do? Validation study guidelines FBI QAS, SWGDAM, etc. Examine the underlying models Consult your neighborhood statistician and molecular biologist about uncertainty and stochastic effects. Google Scholar citations. Accessed 19 February 2016. Available at - https://scholar.google.com/scholar?cites=16343486730052257994

8. Laboratory processes What to do How to do it Do it Did it? Done Developmental validation Internal validation Laboratory protocols Casework Review Report

9. Software engineering “the systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software.” ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.

10. Software development Requirements Specification Design Implementation Testing Maintenance What it does How it does it Source code Software testing Fixes & updates Design Implementation Testing Design Implementation Testing

11. Laboratory protocols Casework Review Report Requirements Specification Design Implementation Maintenance Developmental validation Internal validation Testing Software Lab Matching game These terms are not interchangeable

12. Matching game Requirements Specification Design Implementation Testing Maintenance Software Lab Design Implementation Testing Laboratory protocols Casework

13. Implementation Open source: Panacea? “Open source software is highly desirable in the court environment because openness to scrutiny by any interested party is an invaluable source of bug reports and suggestions for improvement.” Steele, Christopher D., and David J. Balding. "Statistical evaluation of forensic DNA profile evidence." Annual Review of Statistics and Its Application 1 (2014): 361-384. Source code

14. Case studies Therac-25 Radiation therapy machine Killed 6 patients in 1980s by radiation overdosing (software errors + negligence) Classic case study in software engineering Alcotest 7110 Mk III-C Breath alcohol analyzer software Subjected to code review/testing

15. Therac-25 “A ‘small amount’ of software testing was done…” Users were not software experts Oversight and regulation were reactive, not proactive Proprietary claim to software design Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26(7), 18–41.

16. Therac-25 “The materials submitted by the manufacturer have not been in sufficient detail and clarity to ensure an adequate software QA program currently exists.” Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26(7), 18–41.

17. Alcotest 7110 Mk III-C “Key Findings- Testing the Alcotest 7110 Mk III Source Code uncovered 24 major defects.” “Alcotest software would not pass U.S. Industry Standards for Software Development and Testing” “Proof of incomplete software testing” Wisniewski, John. Report: Alcotest 7110 Mk III C. Base One Technologies. Accessed 15 March 2015. Available at - https://www.dwi.com/files/2012/04/1174-base-one-findings.pdf

18. Alcotest 7110 MKIII-C breath alcohol analyzer system “Catastrophic Error Detection is Disabled” “Diagnostics Adjust/Substitute Data Readings” “Error Detection Logic” Wisniewski, John. Report: Alcotest 7110 Mk III C. Base One Technologies. Accessed 15 March 2015. Available at - https://www.dwi.com/files/2012/04/1174-base-one-findings.pdf

19. Acknowledgements Special thanks to: David Bentley Jennifer Friedman Pam King Jennifer Mnookin Michael Risinger Bill Thompson

20. Black Boxes and Due Process: Transparency in Expert Software Systems AAFS 2016 By Nathan Adams – Adams@Bioforensics.com Dan Krane – Dan.Krane@Wright.edu Available at – www.bioforensics.comAdams@Bioforensics.comDan.Krane@Wright.eduwww.bioforensics.com