CCI Firearms and Toolmark Examiner Academy Workshop on Current Firearms and Toolmark Research Pushing Out the Frontiers of Forensic Science

Outline Morning-ish Introduction and the Daubert Standard Confocal Microscopy Focus Variation Microscopy Interferometric Microscopy Surface Data/Filtering

Outline Afternoon-ish Similarity scores and Cross-correlation functions Known Match/Known Non-Match Similarity Score histograms. False Positives/False Negatives/Error Rates Multivariate Discrimination of Toolmarks Measures of “Match Quality” Confidence Posterior Error Rate/Random Match Probability Lessons learned in conducting a successful research project

Introduction DNA profiling the most successful application of statistics in forensic science. Responsible for current interest in “raising standards” of other branches in forensics…?? No protocols for the application of statistics to comparison of tool marks. Our goal: application of objective, numerical computational pattern comparison to tool marks Caution: Statistics is not a panacea!!!!

Daubert (1993)- Judges are the “gatekeepers” of scientific evidence. Must determine if the science is reliable Has empirical testing been done? Falsifiability Has the science been subject to peer review? Are there known error rates? Is there general acceptance? Federal Government and 26(-ish) States are “Daubert States” The Daubert Standard

Tool Mark Comparison Microscope

G. Petillo 4 mm

Known Match Comparisons 5/8” Consecutively manufactured chisels G. Petillo

Known NON Match Comparisons 5/8” Consecutively manufactured chisels G. Petillo

4 mm 600 um 5/8” Consecutively manufactured chisels

Marvin Minsky First confocal microscope Confocal Microscope

Confocal Microscopes

In focus light Out of focus light Tool mark surface (profile of a striation pattern) Focal plane for objective Sample stage Objective lens Illumination aperture Source Confocal pinhole Detector

Rastering pattern of laser confocal Nipkow disk sweeps many pinholes

Programmable array Illumination/Detection Get any illumination/detection pattern

Sample stage Scan stage in “z”-direction Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Sample stage Scan stage in “z”-direction Detector Objective’s focal plane

Detector For Each Detector Pixel: Record the “axial response” as stage is moved along the z-direction Point on surface corresponding to pixel’s is in maximum focus here

Increasing surface height All-in-Focus 2D Image Overlay confocal “z-stack”

3D confocal image of portion of chisel striation pattern

Use high NA objectives for best results Small working distances Flanks up to ~ 70 o Cost ~150K – 250K (FTI IBIS ~1M) Get a vibration isolation table for your instrument ~7K Set up in a (dry) basement if possible Accuracy down to +/- 10 nm Confocal Microscope Trivia Optical slice thickness =

Some manufactures: Olympus LEXT (Laser) Zeiss CSM (White Light) LSM (Laser) Nanofocus  surf series (White Light) Sensofar/Leica Plu series/DCM (White Light) Confocal Microscope Trivia

Focus Variation Microscope Scherer and Prantl “Low res” common Focus variation mic ~ +/- 1  m

In focus light Out of focus light Tool mark surface (profile of a striation pattern) Focal plane for objective Sample stage Objective lens Source Detector

Cutaway Alicona, GMBH

Sample stage Scan stage in “z”-direction Objective’s focal plane

Detector For Each Detector Pixel: Record the “axial response” as stage is moved along the z-direction Point on surface corresponding to pixel is in maximum focus here

Focus Determination: Detector Pixel of interest Compute standard deviation (sd) of pixels grey values in the neighborhood A pixel in focus sits in a neighborhood with a large sd

Use high NA objectives for best results Can use external light Large working distances Flanks up to ~75 o Cost ~200K – 250K. 80K models WON’T have the vertical resolution needed for forensic work Get a vibration isolation table for your instrument ~7K Set up in a (dry) basement if possible Accuracy down to +/- 10nm Focus Variation Microscope Trivia

Some manufactures: Alicona IFM Can get optional rotational stage Sensofar/Leica S neox/DCM Focus Variation Microscope Trivia

Interferometer Incoming wave split Path lengths equal Recombine in-phase Fixed mirror Movable mirror recombine

Interferometer Incoming wave split Path lengths NOT equal Recombine out-of-phase Fixed mirror Movable mirror recombine

Interferometric Height Measurement The basic idea: Each surface point is a “fixed mirror” Move a reference mirror in objective Split beams recombine in and out of phase Constructive interference occurs when surface points in focal plane Infer the surface heights from where constructive interference occurs

Interferometric Microscope James Wyant Early Interferometric Microscope Early Interferometric Microscope for Surafce Metrology Wyant Modern Interferometric Microscope for Surafce Metrology

Tool mark surface (profile of a striation pattern) Focal plane for objective Sample stage Objective lens Camera (Detector) Source Microscope Configuration Piezo Reference mirror Beam-splitter Scan objective for Interference in “z”-direction Path lengths equal Point in focus

Tool mark surface (profile of a striation pattern) Sample stage Objective lens Camera (Detector) Source Microscope Configuration Piezo Reference mirror Beam-splitter Scan objective for Interference in “z”-direction Path lengths un-equal Point in out of focus Focal plane for objective

Interference Objectives Mirau objective ~ 10X – 100X Michelson objective ~ 2X – 10X Linnik objective + 100X

Detector For Each Detector Pixel: Record each pixels interference pattern as objective is scanned Point on surface corresponding To pixel’s is in maximum focus here

Inference patterns: Sample stage Scan objective for Interference in “z”-direction

Fringes Bruker NSD Fringe PatternSurface

Turn Fringes Into A Surface Intensity for each detector pixel: Fourier transform I(z) to get q(k) Compute surface heights deGroot k arg[q(k)] k0k0  A with:

Interferometry Trivia Use high NA objectives for best results Small working distances Flanks up to ~25 o Cost ~200K – 250K. Get a vibration isolation table for your instrument ~7K Set up in a (dry) basement if possible Comes in two modes VSI: Accuracy +/- 10nm PSI: Accuracy below 1nm

Some manufactures: Bruker (Acquired WYKO/Veeco) Taylor Hobson Sensofar/Leica S neox/DCM Interferometry Trivia

Surface Data Surface heights (  m) Land Engraved Area: Point are “double precision”: 64-bits/point BIG FILES!

Surface Data Detector levels (16-bit values): Land Engraved Area: Point are detector grey levels: 16-bits/point Smaller files. Convert to  m in RAM

Different systems use different storage formats Be aware if writing custom apps. ASK COMPANY FOR FILE FORMAT! Alicona: Saves surface data as doubles. HUGE FILES! Zeiss: Saves surface data as 16-bit grey levels with conversion factor Other?? 24, 32-bit detectors now?? Need to standardize file format! X3D Zhang,Brubaker Digital-Surf.sur Petraco Surface Data Trivia

Think of a toolmark surface as being made up of a series of waves Surface Filtering

Examine different scales by “blocking out” (filtering) some of the sinusoids Surface Filtering “Low Pass” filter blocks high frequencies and passes low frequencies (long wavelengths)

Examine different scales by “blocking out” (filtering) some of the sinusoids Surface Filtering “High Pass” filter blocks low frequencies and passes high frequencies (short wavelengths)

Wavelength “cutoffs” Surface Filtering Trivia A “High Pass” filter A “Low Pass” filter cut Wavelength ranges Short wavelengths passed: roughness Medium wavelengths passed: waviness Long wavelengths passed: form

Band-pass filter: Select narrow wavelength bands to keep. High-pass/Low-pass combinations (Filter banks) Wavelets are great at doing this Surface Filtering

Statistics Weapon Mark Association – What measurement techniques can be used to obtain data for toolmarks? – What statistical methods should be used? How do we measure a degree of confidence for an association, i.e. a “match”? What are the identification error rates for different methods of identification?

R is not a black box! Codes available for review; totally transparent! R maintained by a professional group of statisticians, and computational scientists From very simple to state-of-the-art procedures available Very good graphics for exhibits and papers R is extensible (it is a full scripting language) Coding/syntax similar to MATLAB Easy to link to C/C++ routines Why ?

Where to get information on R : R: Just need the base RStudio: A great IDE for R Work on all platforms Sometimes slows down performance… CRAN: Library repository for R Click on Search on the left of the website to search for package/info on packages Why ?

Finding our way around R/RStudio

Gauge similarity between tool marks with one number Similarity “metric” is a function which measures “sameness” Only requirement: s(A,B) = s(B,A) There are an INFINITE number of ways to measure similarity!! Common Computational Practice Often max CCF is used.

Cross-correlation

KNM can sometimes have high max-ccf… max-ccf: 0.751

Glock primer shear: Each profile ~2+ mm Lag over 2000 units (~0.8 mm) Max CCF distributions Cross-Correlation Scores from “Known Non-Matches” Scores from “Known Matches” We thought: Ehhhhhh…….

Random variables - All measurements have an associated “randomness” component Randomness –patternless, unstructured, typical, total ignorance Chaitin, Claude Multivariate Feature Vectors For an experiment/observation, put many measurements together into a list Collection random variables into a list called a random vector 1.Also called: observation vectors feature vectors

Potential feature vectors for surface metrology Entire surfaces *Surface profiles Surface/profile parameters Surface/profile Fourier transform or wavelet coefficients Translation/rotation/scale invariant surface (image) moments Multivariate Feature Vectors

Mean total profile: Mean waviness profile: Waviness profile Barcode representation

Toolmarks (screwdriver striation profiles) form database Biasotti-Murdock Dictionary Consecutive Matching Striae (CMS)-Space

Some Important Terms Latent Variable: weighted combination of experimental variables into a new “synthetic” variable Also called: scores, components or factors The weights are called loadings Most latent variables we will study are linear combinations between experimental variables and loadings: Dot prod. between obs. vect. and loading vect. gives a score:

PCA: Is a rotation of reference frame Gives new PC directions’ relative importance PC variance Principal Component Analysis

Technically, PCA is an eigenvalue-problem Diagonalize some version of S or R to get a PCs Typically Principal Component Analysis covariance matrix matrix of PC “loadings” matrix of PC variances For a data frame of p variables, there are p possible PCs. s ≅ PC importance, dimension reduction Scores are data projected into space of PCs retained Scores plots, either 2D or 3D

Need a data matrix to do machine learning Setup for Multivariate Analysis Represent as a vector of values {-4.62, -4.60, -4.58,...} Each profile or surface is a row in the data matrix Typical length is ~4000 points/profile 2D surfaces are far longer HIGHLY REDUNDANT representation of surface data PCA can: Remove much of the redundancy Make discrimination computations far more tractable

How many PCs should we use to represent the data?? No unique answer FIRST we need an algorithm to I.D. a toolmark to a tool ~45% variance retained 3D PCA of 1740 real and simulated mean profiles of striation patterns from 58 screwdrivers :

Support Vector Machines Support Vector Machines (SVM) determine efficient association rules In the absence of specific knowledge of probability densities SVM decision boundary

Support Vector Machines SVM computed as optimization of “Lagrange multipliers” Quadratic optimization problem Convex => SVMs unique unlike NNs k(x i,x j ) kernel function “Warps” data space and helps to find separations Many forms depending on application: linear, rbf usually C: penalty parameter control the margin of error between groups that are not perfectly separable: 0.1 to 10 usually

Support Vector Machines The SVM decision rule is given as: Equation for a plane in “kernel space” Multi group classification handled by “voting”

How many Principal Components should we use? PCA-SVM With 7 PCs, expect ~3% error rate With 13 PCs, expect ~1% error rate

This supervised technique is called Linear Discriminant Analysis (LDA) in R Also called Fisher linear discriminant analysis CVA is closely related to linear Bayes-Gaussian discriminant analysis Canonical Variate Analysis Works on a principle similar to PCA: Look for “interesting directions in data space” CVA: Find directions in space which best separate groups. Technically: find directions which maximize ratio of between group to within variation

Canonical Variate Analysis Project on PC1: Not necessarily good group separation! Project on CV1: Good group separation! Note: There are #groups -1 or p CVs which ever is smaller

Use between-group to within-group covariance matrix, W -1 B to find directions of best group separation (CVA loadings, A cv ): Canonical Variate Analysis CVA can be used for dimension reduction. Caution! These “dimensions” are not at right angles (i.e. not orthogonal) CVA plots can thus be distorted from reality Always check loading angles! Caution! CVA will not work well with very correlated data

Distance metric used in CVA to assign group i.d. of an unknown data point: If data is Gaussian and group covariance structures are the same then CVA classification is the same as Bayes-Gaussian classification. Canonical Variate Analysis

2D/3D-CVA scores plots of RB screwdrivers 2D CVA3D CVA Canonical Variate Analysis

2D scores plots of RB screwdrivers: PCA vs. CVA 2D PCA of striation pattern mean profiles2D CVA of striation pattern mean profiles

Discriminant functions are trained on a finite set of data How much fitting should we do? What should the model’s dimension be? Error Rate Estimation Model must be used to identify a piece of evidence (data) it was not trained with. Accurate estimates for error rates of decision model are critical in forensic science applications. The simplest is apparent error rate: Error rate on training set Lousy estimate, but better than nothing

Cross-Validation: hold-out chunks of data set for testing Known since 1940s Most common: Hold-one-out Error Rate Estimation Bootstrap: Randomly selection of observed data (with replacement) Known since the 1970s Can yield confidence intervals around error rate estimate The Best: Small training set, BIG test set

Refined bootstrapped I.D. error rate for primer shear striation patterns= 0.35% 95% C.I. = [0%, 0.83%] (sample size = 720 real and simulated profiles) 18D PCA-SVM Primer Shear I.D. Model, 2000 Bootstrap Resamples

How good of a “match” is it? Conformal Prediction Vovk Data should be IID but that’s it Cumulative # of Errors Sequence of Unk Obs Vects 80% confidence 20% error Slope = % confidence 5% error Slope = % confidence 1% error Slope = 0.01 Can give a judge or jury an easy to understand measure of reliability of classification result This is an orthodox “frequentist” approach Roots in Algorithmic Information Theory Confidence on a scale of 0%-100% Testable claim: Long run I.D. error- rate should be the chosen significance level

How Conformal Prediction works for us Given a “bag” of obs with known identities and one obs of unknown identity Vovk Estimate how “wrong” labelings are for each observation with a non- conformity score (“wrong-iness”) Looking at the “wrong-iness” of known observations in the bag: Does labeling-i for the unknown have an unusual amount of “wrong-iness”??: For us, one-vs-one SVMs: If not: p possible-ID i ≥ chosen level of significance Put ID i in the (1 - )*100% confidence interval

Conformal Prediction Theoretical (Long Run) Error Rate: 5% Empirical Error Rate: 5.3% 14D PCA-SVM Decision Model for screwdriver striation patterns For 95%-CPT (PCA-SVM) confidence intervals will not contain the correct I.D. 5% of the time in the long run Straight-forward validation/explanation picture for court

Conformal Prediction Drawbacks CPT is an interval method Can (and does) produce multi-label I.D. intervals A “correct” I.D. is an interval with all labels Doesn’t happen often in practice… Empty intervals count as “errors” Well…, what if the “correct” answer isn’t in the database An “Open-set” problem which Champod, Gantz and Saunders have pointed out Must be run in “on-line” mode for LRG After 500+ I.D. attempts run in “off-line” mode we noticed in practice

An I.D. is output for each questioned toolmark This is a computer “match” What’s the probability it is truly not a “match”? Similar problem in genomics for detecting disease from microarray data They use data and Bayes’ theorem to get an estimate No disease genomics = Not a true “match” toolmarks How good of a “match” is it? Efron Empirical Bayes’

Empirical Bayes’ We use Efron’s machinery for “empirical Bayes’ two-groups model” Efron Surprisingly simple! Use binned data to do a Poisson regression Some notation: S -, truly no association, Null hypothesis S +, truly an association, Non-null hypothesis z, a score derived from a machine learning task to I.D. an unknown pattern with a group z is a Gaussian random variate for the Null

Empirical Bayes’ From Bayes’ Theorem we can get Efron : Estimated probability of not a true “match” given the algorithms' output z-score associated with its “match” Names: Posterior error probability (PEP) Kall Local false discovery rate (lfdr) Efron Suggested interpretation for casework: We agree with Gelaman and Shalizi Gelman : = Estimated “believability” of machine made association “…posterior model probabilities …[are]… useful as tools for prediction and for understanding structure in data, as long as these probabilities are not taken too seriously.”

Empirical Bayes’ Bootstrap procedure to get estimate of the KNM distribution of “Platt-scores” Platt,e1071 Use a “Training” set Use this to get p-values/z-values on a “Validation” set Inspired by Storey and Tibshirani’s Null estimation method Storey z-score From fit histogram by Efron’s method get: “mixture” density We can test the fits to and ! What’s the point?? z-density given KNM => Should be Gaussian Estimate of prior for KNM Use SVM to get KM and KNM “Platt-score” distributions Use a “Validation” set

Posterior Association Probability: Believability Curve 12D PCA-SVM locfdr fit for Glock primer shear patterns +/- 2 standard errors

Bayesian over-dispersed Poisson with intercept on test setBayesian Poisson with intercept on test set Poisson (Efron) on test set Bayesian Poisson on test set

Bayes Factors/Likelihood Ratios In the “Forensic Bayesian Framework”, the Likelihood Ratio is the measure of the weight of evidence. LRs are called Bayes Factors by most statistician LRs give the measure of support the “evidence” lends to the “prosecution hypothesis” vs. the “defense hypothesis” From Bayes Theorem:

Bayes Factors/Likelihood Ratios Once the “fits” for the Empirical Bayes method are obtained, it is easy to compute the corresponding likelihood ratios. o Using the identity: the likelihood ratio can be computed as:

Bayes Factors/Likelihood Ratios Using the fit posteriors and priors we can obtain the likelihood ratios Tippett, Ramos Known match LR values Known non-match LR values

Empirical Bayes’: Some Things That Bother Me Need a lot of z-scores Big data sets in forensic science largely don’t exist z-scores should be fairly independent Especially necessary for interval estimates around lfdr Efron Requires “binning” in arbitrary number of intervals Also suffers from the “Open-set” problem Interpretation of the prior probability for this application Should Pr(S - ) be 1 or very close to it? How close?

How to Carry Out a “Successful” Research Project The Synergy Between Practitioners and Academia

Collaboration Practitioners: Think about what questions you want to be able to answer with data BEFORE experimentation Write down proposed questions/design Be aware that the questions you want answers too MAY NOT have answers What can you answer?? Be aware that a typical research project takes 1-2 years to complete

Collaboration Practitioners: Research projects are NOT just for interns! Interns typically need tremendous supervision for scientific/applied statistical research Take a college course on statistics/experimental design Rate-my-professor is your friend! Visit local university/company websites to look for the outside expertise you may need. Visit the department, go to some seminars

Collaboration Academics/Research consultants: Be aware practitioners cannot just publish whenever and whatever they want Long internal review processes! COMMUNICATION!!!!! Listen carefully to the needs/questions of collaborating practitioners Negotiate the project design What kind of results can be achieved within a reasonable amount of time? Hold regular face to face meetings if possible

Collaboration Academics/Research consultants: Applied research is not just for undergraduates/high-school interns! Visit the crime lab!!!!! Watch the practitioners do their job. Learn the tools they use day to day! Microscopy!!!!! Use their accumulated experience to help guide your design/desired outcomes What do they focus on??

Fire Debris Analysis Casework Liquid gasoline samples recovered during investigation: Unknown history Subjected to various real world conditions. If an individual sample can be discriminated from the larger group, this can be of forensic interest. Gas-Chromatography Commonly Used to ID gas. Peak comparisons of chromatograms difficult and time consuming. Does “eye-balling” satisfy Daubert, or even Frye.....????

2D PCA 97.3% variance retained Avg. LDA HOO correct classification rate: 83%

2D CVA Avg. LDA HOO correct classification rate: 92%

Accidental Patterns on Footwear Shoe prints contain marks and patterns due to various circumstances that can be used to distinguish one shoe print from another. How reliable are the accidental patterns for identifying particular shoes?

3D PCA 59.7% of variance Facial Recognition Approach to Accidental Pattern Identification

Tool marks Like shoes, tools can leave marks which can be used in identification Class characteristics Subclass characteristics Individual characteristics

Standard Striation Patterns Made with ¼’’ Slotted Screwdriver Measure lines and grooves with ImageJ Translate ImageJ data to a feature vector that can be processed

A, 2, #2 Bromberg, Lucky C, 8, #4 Bromberg, Lucky LEA Striations

Questioned Documents: Photocopier Identification Mordente, Gestring, Tytell Photocopy of a blank sheet of paper

Dust: Where does it come from? Any matter or substance: both natural and synthetic reduces into minute bits, pieces, smears, and residues encountered as trace aggregates Our Environments! Evidence! N. Petraco

Where can you find it?Everywhere House Work Outdoors Vehicle N. Petraco

Analyze Results 3D PCA-Clustering can show potential for discrimination

Bayes Net for Dust in Authentication Case

References Bolton-King, Evans, Smith, Painter, Allsop, Cranton. AFTE J 42(1), Artigas. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l Helmli. In: Optical Measurement of Surface Topography. Leach ed. Springer, 2011 deGroot. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l Efron, B. (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. New York: Cambridge University Press. Gambino C., McLaughlin P., Kuo L., Kammerman F., Shenkin S., Diaczuk P., Petraco N., Hamby J. and Petraco N.D.K., “Forensic Surface Metrology: Tool Mark Evidence", Scanning 27(1-3), 1-7 (2011). JAGS “A program for analysis of Bayesian hierarchical models using Markov Chain Monte Carlo simulation”, Version Kall L., Storey J. D., MacCross M. J. and Noble W. S. (2008). Posterior error probabilities and false discovery rates: two sides of the same coin. J Proteome Research, 7(1),

References locfdr R package locfdr: “Computation of local false discovery rates”, Version Moran B., "A Report on the AFTE Theory of Identification and Range of Conclusions for Tool Mark Identification and Resulting Approaches To Casework," AFTE Journal, Vol. 34, No. 2, 2002, pp Petraco a N. D. K., Chan H., De Forest P. R., Diaczuk P., Gambino C., Hamby J., Kammerman F., Kammrath B. W., Kubic T. A., Kuo L., Mc Laughlin P., Petillo G., Petraco N., Phelps E., Pizzola P. A., Purcell D. K. and Shenkin P. “Final Report: Application of Machine Learning to Toolmarks: Statistically Based Methods for Impression Pattern Comparisons”. National Institute of Justice, Grant Report: 2009-DN-BX-K041; Petraco N. D. K., Kuo L., Chan H., Phelps E., Gambino C., McLaughlin P., Kammerman F., Diaczuk P., Shenkin P., Petraco N. and Hamby J. “Estimates of Striation Pattern Identification Error Rates by Algorithmic Methods”, AFTE J., In Press, Petraco N. D. K., Zoon P., Baiker M., Kammerman F., Gambino C. “Stochastic and Deterministic Striation Pattern Simulation”. In preparation Platt J. C. “Probabilities for SV Machines”. In: Advances in Large Margin Classifiers Eds: Smola A. J., Bartlett P., Scholkopf B., and Schuurmans D. MIT Press, Plummer M. “JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling”, Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), March 20– 22, Vienna, Austria.“JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling Stan Development Team “Stan: A C++ Library for Probability and Sampling”, Version stan.org/ stan.org/ Storey J. D. and Tibshirani R. “Statistical significance for genome wide studies”. PNAS 2003;100(16): Vovk V., Gammerman A., and Shafer G. (2005). Algorithmic learning in a random world. 1st ed. Springer, New York.

References 20.Tippett CF, Emerson VJ, Fereday MJ, Lawton F, Richardson A, Jones LT, Lampert SM., “The Evidential Value of the Comparison of Paint Flakes from Sources other than Vehicles”, J Forensic Soc Soc 1968 ;8(2-3): Ramos D, Gonzalez-Rodriguez J, Zadora G, Aitken C. “Information-Theoretical Assessment of the Performance of Likelihood Ratio Computation Methods”, J Forensic Sci 2013;58(6):

Acknowledgements Professor Chris Saunders (SDSU) Professor Christoph Champod (Lausanne) Alan Zheng (NIST) Research Team: Dr. Martin Baiker Ms. Helen Chan Ms. Julie Cohen Mr. Peter Diaczuk Dr. Peter De Forest Mr. Antonio Del Valle Ms. Carol Gambino Dr. James Hamby Ms. Alison Hartwell, Esq. Dr. Thomas Kubic, Esq. Ms. Loretta Kuo Ms. Frani Kammerman Dr. Brooke Kammrath Mr. Chris Lucky Off. Patrick McLaughlin Dr. Linton Mohammed Mr. John Murdock Mr. Nicholas Petraco Dr. Dale Purcel Ms. Stephanie Pollut Dr. Peter Pizzola Dr. Graham Rankin Dr. Jacqueline Speir Dr. Peter Shenkin Mr. Chris Singh Mr. Peter Tytell Mr. Todd Weller Ms. Elizabeth Willie Dr. Peter Zoon

Website: Data, codes, reprints and preprints: toolmarkstatistics.no-ip.org/