Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Application of statistical methods for the comparison of data distributions Susanna Guatelli, Barbara Mascialino, Andreas Pfeiffer, Maria Grazia Pia, Alberto Ribon, Paolo Viarengo

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 The comparison of two data distribution is fundamental in experimental practice Many algorithms are available for the comparison of two data distributions (the two-sample problem) Aim of this study: Aim of this study: compare the algorithms available in statistics literature to select the most appropriate one in every specific case Outline Detector monitoring Detector monitoring (current versus reference data) Simulation validation (experiment versus simulation) Reconstruction versus expectation Regression testing (two versions of the same software) Physics analysis Physics analysis (measurement versus theory, experiment A versus experiment B) Parametric statistics Non-parametric statistics (Goodness-of-Fit testing)

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 The two-sample problem EXAMPLE 1 EXAMPLE 1: binned data Which is the most suitable goodness-of-fit test? EXAMPLE 2 EXAMPLE 2: unbinned data X-ray fluorescence spectrum Dosimetric distribution from a medical LINAC

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 binnedApplies to binned distributions It can be useful also in case of unbinned distributions, but the data must be grouped into classes Cannot be applied if the counting of the theoretical frequencies in each class is < 5 –When this is not the case, one could try to unify contiguous classes until the minimum theoretical frequency is reached –Otherwise one could use Yates’ formula Chi-squared test

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS Kolmogorov-Smirnov test Goodman approximation of KS test Kuiper test D mn Tests based on the supremum statistics unbinned distributions SUPREMUMSTATISTICS

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Fisz-Cramer-von Mises test k-sample Anderson-Darling test Tests containing a weighting function binned/unbinned distributions EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS QUADRATICSTATISTICS+ WEIGHTING FUNCTION Sum/integral of all the distances

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): October issue.

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Power evaluation N=1000 Monte Carlo replications Confidence Level = 0.05 Pseudoexperiment: a random drawing of two samples from two parent distributions For each test, the p-value computed by the GoF Toolkit derives from analytical calculation of the asymptotic distribution, often depending on the samples sizes. The power of a test is the probability of rejecting the null hypothesis correctly Parent distribution 1 Sample 1 n Sample 2 m GoF test Parent distribution 2 Power Power = # pseudoexperiments with p-value < (1-CL) # pseudoexperiments

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Parent distributions Uniform Gaussian Double exponential Cauchy Exponential Contaminated Normal Distribution 2Contaminated Normal Distribution 1

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Skewness and tailweightParentST f 1 (x) Uniform f 2 (x) Gaussian f 3 (x) Double exponential f 4 (x) Cauchy f 5 (x) Exponential f 6 (x) Contamined normal f 7 (x) Contamined normal SkewnessTailweight

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Power increases as a function of the sample size (analytical calculation of the asymptotic distribution) N sample Power Kolmogorov-Smirnov test CL = 0.05 The “location-scale problem” Case Parent1 = Parent 2 Uniform Normal Exponential Double Exponential Contaminated Normal 1 Contaminated Normal 2 Cauchy small sized samples moderate sized samples

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 The “general shape problem” Distribution1 – Distribution 2KSCVMAD CN2-Normal55.6± ± ±1.1 CN2-CN124.9± ± ±1.6 CN2-Double Exponential37.6± ± ±1.6 T2T2 Case Parent1 ≠ Parent 2 Power Tailweight Distribution 2 CL = 0.05 Kolmogorov-Smirnov Cramér-von Mises Anderson-Darling (S 1 = S 2 = 1) Distribution 1 Double exponential (T 1 = 2.161) A) Symmetric versus symmetric B) Skewed versus symmetric KSKSCVMCVMADAD ~< For very long tailed distributions: KSKSCVMCVMADAD ~~ For short-medium tailed distributions:

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Comparative evaluation of tests Short (T<1.5) Medium (1.5 < T < 2) Long(T>2) S~1S~1S~1S~1KS KS – CVM CVM - AD S>1.5 KS - AD AD CVM - AD Skewness Tailweight 2222 2222 Supremum statistics tests Tests containing a weight function < <

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Results for the data examples Extremely skewed – medium tail ANDERSON-DARLING TEST A 2 =0.085 – p>0.05 Moderate skewed – medium tail KOLMOGOROV-SMIRNOV TEST D=0.27 – p>0.05 X-variable: Ŝ=4 T=1.43 Y-variable: Ŝ=4 T=1.50 X-variable: Ŝ=1.53 T=1.36 Y-variable: Ŝ=1.27 T=1.34 ^ ^ ^ ^ EXAMPLE 1 EXAMPLE 1: binned data EXAMPLE 2 EXAMPLE 2: unbinned data

Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Studied several goodness-of-fit tests for location-scale alternatives and general alternatives noThere is no clear winner for all the considered distributions in general To select one test in practice: 1.classify ST 1. first classify the type of the distributions in terms of skewness S and tailweight T 2.most 2. choose the most appropriate test for the classified type of distribution Conclusions Topic still subject to research activity in the domain of statistics