Alberto Ribon, CERN Statistical Testing Project Alberto Ribon, CERN on behalf of the Statistical Testing Team CLHEP Workshop CERN, 28 January 2003

Alberto Ribon, CERN What is? Provide tools for the statistical comparison of distributions – simulation data – experimental measurements – data from reference sources – functions deriving from theoretical calculations or from fits physics validation regression testing system testing Main application areas in Geant4: A project to develop a general purpose statistical analysis system A project to develop a general purpose statistical analysis system

Alberto Ribon, CERN The team Development team (mostly part time!) Pablo Cirrone, INFN Southern National Lab Stefania Donadio, Univ. and INFN Genova Susanna Guatelli, CERN/IT/API Technical Student and INFN Genova Alberto Lemut, Univ. and INFN Genova Barbara Mascialino, Univ. and INFN Genova Sandra Parlati, INFN Gran Sasso National Lab Andreas Pfeiffer, CERN/IT/API Maria Grazia Pia, INFN Genova Alberto Ribon, CERN/IT/API Statistical consultancy Paolo Viarengo, Univ. Genova, Statistician Fred James, CERN Geant4 system integration team Gabriele Cosmo, CERN/IT/API - Geant4 Release Manager Sergei Sadilov, CERN/IT/API - Geant4 System Testing Coordinator interested collaborators are welcome!

Alberto Ribon, CERN Scope of the project tools for statistical testing The project will provide tools for statistical testing –physics comparisons and regression testing –multiple comparison algorithms Generality Generality (for application also in other areas) should be pursued –facilitated by a component-based architecture The statistical tools should be used in Geant4 (and in other frameworks) –tool to be used in testing frameworks –not a testing framework itself Re-use existing tools whenever possible –no attempt to re-invent the wheel –but critical, scientific evaluation of candidate tools

Alberto Ribon, CERN So far, only ad hoc solutions An old and common problem (comparison of distributions) The only general “tool” was HDIFF (which does the Kolmogorov- Smirnov test), which, although very useful and used, was never enough for any realistic physics analysis Each experiment (or even each Analysis group) has created each time its ad hoc “tool” for statistical tests, usually based on legacy code which were modified and adapted for the particular needs Example: CDF Coll. PRL 77 (1996) 438 “Inclusive jet cross section in p-pbar collisions at Tevatron”

Alberto Ribon, CERN Architectural guidelines architectural The project adopts a solid architectural approach functionalityquality –to offer the functionality and the quality needed by the users maintainable –to be maintainable over a large time scale extensible –to be extensible, to accommodate future evolutions of the requirements Component-based approach –Geant4-specific + general –Geant4-specific components + general components –to facilitate re-use and integration in diverse frameworksAIDA –adopt a (HEP) standard –no dependence on any specific analysis toolPython The approach adopted is compatible with the recommendations of the LCG Architecture Blueprint RTAG CERN LCG Architecture Blueprint RTAG

Alberto Ribon, CERN Some use cases Regression testing –Throughout the software life-cycle Online DAQ –Monitoring detector behaviour w.r.t. a reference Simulation validation –Comparison with experimental data Reconstruction –Comparison of reconstructed vs. expected distributions Physics analysis –Comparisons of experimental distributions (signal sample vs. bkg sample) –Comparison with theoretical distributions (data vs. Standard Model)

Alberto Ribon, CERN Goodness-of-fit tests Pearson’s  2 test Kolmogorov test Kolmogorov – Smirnov test Lilliefors test Cramer-von Mises test Anderson-Darling test Kuiper test … System open to extension and evolution Suggestions welcome!

Alberto Ribon, CERN Pearson’s  2 discrete (binned) Applies to discrete (binned) distributions It can be useful also in case of continuous (unbinned) distributions, but the data must be grouped into classes Cannot be applied if the counting of the theoretical frequencies in each class is < 5 When this is not the case, one could try to unify contiguous classes until the minimum theoretical frequency is reached

Alberto Ribon, CERN Kolmogorov test The easiest among non-parametric tests continuous Verify the adaptation of a sample coming from a random continuous variable Based on the computation of the maximum distance between an empirical repartition function and the theoretical repartition one Test statistics: D = sup | F O (x) - F T (x)|

Alberto Ribon, CERN Kolmogorov-Smirnov test Problem of the two samples –mathematically similar to Kolmogorov’s Instead of comparing an empirical distribution with a theoretical one, try to find the maximum difference between the distributions of the two samples F n and G m : D mn = sup |F n (x) - G m (x)| continuous Can be applied only to continuous random variables Conover (1971) and Gibbons and Chakraborti (1992) tried to extend it to cases of discrete random variables

Alberto Ribon, CERN Lilliefors test Similar to Kolmogorov test Based on the null hypothesis that the random continuous variable is normally distributed N(m,  2 ), with m and  2 unknown Performed comparing the empirical repartition function F(z 1,z 2,...,z n ) with the one of the standardized normal distribution  (z): D* = sup | F O (z) -  (z)|

Alberto Ribon, CERN Cramer-von Mises test Based on the test statistics:  2 = integral (F O (x) - F T (x)) 2 dF(x) continuousdiscrete Can be performed both on continuous and discrete variables Satisfactory for symmetric and right-skewed distributions

Alberto Ribon, CERN Anderson-Darling test Performed on the test statistics: A 2 = integral { [F O (x) – F T (x)] 2 / [F T (x) (1-F T (X))] } dF T (x) continuousdiscrete Can be performed both on continuous and discrete variables skewness Seems to be suitable to any data-set (Aksenov and Savageau ) with any skewness (symmetric distributions, left or right skewed) Seems to be sensitive to fat tail of distributions

Alberto Ribon, CERN Kuiper test Based on a quantity that remains invariant for any shift or re-parameterization Does not work well on tails D* = max (F O (x)-F T (x)) + max (F T (x)-F O (x))

Alberto Ribon, CERN OOAD Collection of user requirements First analysis and design of the statistical component Validation of the class design through use cases Some open issues identified, to be addressed in the next design iterations

Alberto Ribon, CERN + more algorithms

Alberto Ribon, CERN

Work in progress Implementation and test of preliminary design What can be re-used? –Almost nothing available either in GSL or NAG Studies in progress –Transformation between binned-unbinned distributions –Strategies to use Kolmogorov-Smirnov with binned distributions (E. Dagum + original ideas) –How to deal with experimental errors (not only statistical!) –Multi-dimensional distributions –Bayesian approach In the to-do list –Conversion from AIDA objects to distributions –“Pythonisation”

Alberto Ribon, CERN Work in progress: User-specific Geant4 testing framework –Development of general physics tests in E.M. domain: collection of relevant observables, and respective reference data/distributions –Integration in the system testing framework CMS transition from Geant3 to Geant4 –An automaatic regression testing procedure is needed –Similar needs also for future Geant4 versions

Alberto Ribon, CERN Where? Core statistical component –Developed in an independent CVS repository –Code, documentation, software process deliverables –Where it will go? CLHEP or LCG ? Geant4-specific stuff –Kept separated in Geant4 Web site – Contact persons

Alberto Ribon, CERN Time scale driven by User needs Aggressive time scale driven by User needs –CMS and Geant4 OOAD + implementation undergoing A first prototype should be ready in few weeks Advanced functional system summer 2003 Open to the needs/suggestions of anyone –compatible with the available resources –possible integration in GSL

Alberto Ribon, CERN Conclusions… Core statistical components of general interest –LHC experiments, Geant4, etc. Project compatible with LCG architecture blueprint –component-based approach, AIDA, Python… Open to scientific collaboration Urgent user needs –CMS and Geant4 First prototype expected in few weeks