B.Mascialino, A.Pfeiffer, M.G.Pia, A.Ribon, P.Viarengo STATISTICAL TOOLKIT B.Mascialino, A.Pfeiffer, M.G.Pia, A.Ribon, P.Viarengo Geant4 Workshop Catania, October 4th-9th 2004

Goodness-of-Fit testing Provide tools for the statistical comparison of distributions equivalent reference distributions experimental measurements data from reference sources functions deriving from theoretical calculations or fits Qualitative evaluation Quantitative evaluation A project to develop a statistical comparison system Comparison of distributions Goodness of fit testing Detector monitoring Simulation validation Reconstruction vs. expectation Regression testing Physics analysis Detector monitoring in order to check if the behavior is constant in more than one run

Architectural guidelines The project adopts a solid architectural approach to offer the functionality and the quality needed by the users to be maintainable over a large time scale to be extensible, to accommodate future evolutions of the requirements Component-based approach to facilitate re-use and integration in different frameworks AIDA adopt a (HEP) standard no dependence on any specific analysis tool

Software process guidelines United Software Development Process, specifically tailored to the project practical guidance and tools from the RUP both rigorous and lightweight mapping onto ISO 15504 Guidance from ISO 15504 Incremental and iterative life cycle model SPIRAL APPROACH

Requirement traceability User Requirements User requirements elicited, analysed and formally specified Functional (capability) and not-functional (constraint) requirements User Requirements Document available from the web site Requirements Design Implementation Test & test results Documentation Requirement traceability

It is externally distributed with PI The algorithms are specialised on the kind of distribution (binned/unbinned) Every algorithm has been rigorously tested! The Toolkit is downloadable from the web: http://www.ge.infn.it/geant4/analysis/HEPstatistics/ It is externally distributed with PI

Chi-squared test Applies to binned distributions It can be useful also in case of unbinned distributions, but the data must be grouped into classes Cannot be applied if the counting of the theoretical frequencies in each class is < 5 When this is not the case, one could try to unify contiguous classes until the minimum theoretical frequency is reached Otherwise one could use Yates’ formula

More sophisticated algorithms unbinned distributions Kolmogorov-Smirnov test Goodman approximation of KS test Kuiper test EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS Dmn SUPREMUM STATISTICS

More powerful algorithms unbinned distributions Cramer-von Mises test Anderson-Darling test TESTS CONTAINING A WEIGHTING FUNCTION These algorithms are so powerful that we decided to implement their equivalent in case of binned distributions: Fisz-Cramer-von Mises test k-sample Anderson-Darling test

2 Is 2 the most powerful algorithm? In terms of power: The power of a test is the probability of rejecting the null hypothesis correctly In terms of power: 2 Supremum statistics tests Tests containing a weight function < Test Power Characteristics Anderson-Darling High Sensitive to tails c2 Low General Fisz-Cramer-von Mises Symmetric, right-skewed distributions Goodman Medium Approximation of K-S to c2 test statistics Kolmogorov-Smirnov Derives from Kolmogorov statistics Kuiper Sensitive to tails and median Tiku Converts CvM statistics to a c2 Talk at IEEE NSS, Rome, 16-22 October 2004 + paper submitted for publication November 2004

Feedback from users is welcome! GPL License Feedback from users is welcome!

User Documentation Download Installation User Guide Statistics Reference Guide User Documentation

EXTRACTS THE ALGORITHM WRITING ONE LINE OF CODE User’s point of view Simple user layer Only deal with AIDA objects and choice of comparison algorithm The user is completely shielded from both statistical and computing complexity. STATISTICAL RESULT TOOLKIT USER EXTRACTS THE ALGORITHM WRITING ONE LINE OF CODE

Examples of practical applications

THANKS TO SUSANNA GUATELLI Microscopic validation of physics p=1 NIST Geant4 Standard Geant4 LowE Geant4 simulations are statistically comparable with reference data (NIST database http://www.nist.gov) Chi-squared test THANKS TO SUSANNA GUATELLI READY FOR REGRESSION TESTING

X-ray fluorescence spectrum in Iceand basalt Test beam at Bessy Bepi-Colombo mission Energy (keV) Counts X-ray fluorescence spectrum in Iceand basalt (EIN=6.5 keV) Very complex distributions c2 not appropriate (< 5 entries in some bins, physical information would be lost if rebinned) Anderson-Darling p>0.05 THANKS TO ALFONSO MANTERO Experimental measurements are comparable with Geant4 simulations

THANKS TO MICHELA PIERGENTILI Medical physics: IMRT treatment at THANKS TO MICHELA PIERGENTILI Kolmogorov-Smirnov test Distance range Test statistics P-value -84  -60 mm 0.385 0.23 -59  -48 mm 0.27 0.90 -47  47 mm 0.43 0.19 48  59 mm 0.30 0.82 60  84 mm 0.40 0.10 Distance range Test statistics P-value -56  -35 mm 0.26 0.89 -34  -22 mm 0.43 0.42 -21 21 mm 0.38 0.08 22  32 mm 0.98 33  36 mm 0.57 0.13

Conclusions Applications in: HEP, astrophysics, medical physics, … This is a new up-to-date easy to handle and powerful tool for statistical comparison in particle physics. Rigorous software process to contribute to the quality of the product Component-based architecture, OO methods + generic programming to ensure openness to evolution, maintainability, ease of use It the first tool supplying such a variety of sophisticated and powerful statistical tests in HEP. AIDA interfaces allow its integration in any other concrete data analysis tool. Applications in: HEP, astrophysics, medical physics, … WE INVITE ANYONE TO USE IT!!!!

Future developments Power comparison among algorithms Extension to theoretical functions Extensions to bidimensional distributions