An update on the Statistical Toolkit Barbara Mascialino, Maria Grazia Pia, Andreas Pfeiffer, Alberto Ribon, Paolo Viarengo July 19 th, 2005

G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): Release StatisticsTesting-V downloadable from the web:

EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS Kolmogorov-Smirnov test Goodman approximation of KS test Kuiper test D mn Tests based on maximum distance unbinned distributions SUPREMUMSTATISTICS

Fisz-Cramer-von Mises test Anderson-Darling test Tests containing a weighting function binned/unbinned distributions EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS QUADRATICSTATISTICS+ WEIGHTING FUNCTION Sum/integral of all the distances

1.Status of the existing tests 2. New GoF Tests added 3. Description of the power study phase I 4. Description of the power study phase II 5. A concrete example: IMRT

1.Status of the existing tests: Fisz-Cramer-von Mises Conover (book) + Darling (1957): - The two-sample Cramer-von Mises test (Fisz test) has the same asymptotic distribution of the one-sample test (Cramer-von Mises test). - The equation of the asymptotic distribution is available in the paper by Anderson and Darling (1952). binned/unbinned distributions

1.Status of the existing tests: two-sample Anderson-Darling Scholz and Stephens (1987): - -The two-sample Anderson-Darling test can be written in different ways: - exact formulation (for unbinned distributions only) - approximated formulation (for binned/unbinned distributions) - -The approximated distance is already available in the toolkit. - -The asymptotic distributions of both exact and approximated formulations are available in the paper. - - The two-sample Anderson-Darling test has the same asymptotic distribution of the one-sample test. binned/unbinned distributions

1.Status of the existing tests: Tiku test Tiku (1965): - - Cramer-von Mises test in a chi-squared approximation. - - Cramer-von Mises test statistics is converted into a central chi-square, bypassing the problem of integrating the weighting function. binned/unbinned distributions

2. New GoF Tests: weighted Kolmogorov-Smirnov Canner (1975) & Buning (2001): - - Canner modified KS test introducing one weighting function identical to the one used in AD test. - - Buning modified KS test introducing one weighting function similar to the one used in AD test. - The equation of the asymptotic distribution is not available in Canner’s paper, only a few critical values for some samples sizes (n=m). unbinned distributions

2. New GoF Tests: weighted Cramer von Mises Buning (2001): - - Buning modified CVM test introducing one weighting function similar to the one used in AD test. - The equation of the asymptotic distribution is not available in the paper, only critical values for many samples sizes. unbinned distributions 2

2. New GoF Tests: Watson Watson (1975): - -Derives from Cramer-von Mises test statistics. - - Like Kuiper test it can be applied in case of cyclic observations. - The equation of the asymptotic distribution is not available in the paper, only critical values for many samples sizes. 22

Other news New user layer dealing with ROOT histograms (Andreas is working on that). Paper to IEEE-TNS Next release of the GoF Statistical Toolkit scheduled within summer.

Future developments Fix some design-related problems. New design (add uncertainties). Extend the toolkit to the comparison of: –Experimental data versus theoretical functions, –k-sample problem, –Many dimensional one-, two-, k-sample problem.

Which is the recipe to select the most suitable Goodness-of-Fit test among the ones available in the GoF Statistical Toolkit?

3. Description of the power study – phase I SAMPLE 1 RESULTS:LOCATION-SCALEALTERNATIVE SAMPLE 2 TEST PARENT 1 PARENT 2 MONTECARLOREPLICATIONSk=1000 EDF STATISTICS (UNBINNED DATA): KS, KSW, KSA, KUIPER, CVM, ADA “EMPIRICAL” POWER EVALUATION RESULTS:GENERALALTERNATIVE COMPARISON WITH PUBLISHEDRESULTSREALDATAEXAMPLES

Parent distributions Uniform Gaussian Double exponential Cauchy Exponential Contaminated Normal Distribution 2Contaminated Normal Distribution 1

Skewness and tailweight ParentST f 1 (x) Uniform f 2 (x) Gaussian f 3 (x) Double exponential f 4 (x) Cauchy f 5 (x) Exponential f 6 (x) Contamined normal f 7 (x) Contamined normal SkewnessTailweight

Comparative evaluation of tests Short (T<1.5) Medium (1.5 < T < 2) Long(T>2) S~1S~1S~1S~1KS KS – CVM CVM - AD S>1.5 KS - AD AD CVM - AD Skewness Tailweight 2222 2222 Supremum statistics tests Tests containing a weight function < <

4. Description of the power study – phase II SAMPLE 1 RESULTS:LOCATION-SCALEALTERNATIVE SAMPLE 2 TEST PARENT 1 PARENT 2 MONTECARLOREPLICATIONSk=1000 BINNED/UNBINNED DATA CHI2, KS, KSW, KSA, KUIPER, CVM, CVMA, ADA, AD “EMPIRICAL” + “MC” POWER EVALUATION RESULTS:GENERALALTERNATIVELINEARPOWERCORRELATIONBETWEENTESTSISODYNES

Which is the most suitable goodness-of-fit test? EXAMPLE EXAMPLE : unbinned data Lateral profiles 5. A concrete example: IMRT MichelaPiergentili

GoF test selection SkewnessTailweight S = 1: symmetric distribution S < 1: left skewed distribution S > 1: right skewed distribution T is always greater than 1, the longer the tail the greater the value of T. 1.Classify the type of the distributions in terms of skewness S and tailweight T

Comparative evaluation of tests power Short (T<1.3) Medium (1.3 < T < 2) Long(T>2) S~1S~1S~1S~1KS KS – CVM CVM - AD S>1.5 KS - AD AD CVM - AD Skewness Tailweight 2. Choose the most appropriate test for the classified type of distribution

GoF test: test selection & results Moderate skewed – medium tail KOLMOGOROV-SMIRNOV TEST D=0.27 – p>0.05 X-variable: Ŝ=1.53 T=1.36 Y-variable: Ŝ=1.27 T=1.34 ^ ^ RESULTS RESULTS: unbinned data