Maria Grazia Pia, INFN Genova Statistical Testing Project Maria Grazia Pia, INFN Genova on behalf of the Statistical Testing Team

Slides:



Advertisements
Similar presentations
Alberto Ribon CERN Geant4Workshop Vancouver, September 2003 Tutorial of the Statistical Toolkit
Advertisements

Statistical Toolkit Power of Goodness-of-Fit tests
Physicist Interfaces Project an overview Physicist Interfaces Project an overview Jakub T. Moscicki CERN June 2003.
Maria Grazia Pia, INFN Genova Test & Analysis Project Maria Grazia Pia, INFN Genova on behalf of the T&A team
Configuration management
Configuration management
Precision validation of Geant4 electromagnetic physics Katsuya Amako, Susanna Guatelli, Vladimir Ivanchenko, Michel Maire, Barbara Mascialino, Koichi Murakami,
Maria Grazia Pia Geant4 LowE Workshop 30-31/5/2002 ow Energy e.m. Workshop CERN, May 2002.
1 Prescriptive Process Models. 2 Prescriptive Models Prescriptive process models advocate an orderly approach to software engineering Prescriptive process.
Maria Grazia Pia, INFN Genova 1 Part V The lesson learned Summary and conclusions.
Barbara MascialinoIEEE-NSSOctober 21 th, 2004 Application of statistical methods for the comparison of data distributions Susanna Guatelli, Barbara Mascialino,
Rational Unified Process
Barbara Mascialino, INFN Genova An update on the Goodness of Fit Statistical Toolkit B. Mascialino, A. Pfeiffer, M.G. Pia, A. Ribon, P. Viarengo
Iterative development and The Unified process
Maria Grazia Pia, INFN Genova A Toolkit for Statistical Data Analysis M.G. Pia S. Donadio, F. Fabozzi, L. Lista, S. Guatelli, B. Mascialino, A. Pfeiffer,
Maria Grazia Pia, INFN Genova Test & Analysis Project Maria Grazia Pia, INFN Genova on behalf of the T&A team
Comparison of data distributions: the power of Goodness-of-Fit Tests
UML - Development Process 1 Software Development Process Using UML (2)
Introduction to RUP Spring Sharif Univ. of Tech.2 Outlines What is RUP? RUP Phases –Inception –Elaboration –Construction –Transition.
Maria Grazia Pia, INFN Genova Software Process: Physics Maria Grazia Pia INFN Genova on behalf of the Geant4 Collaboration Budker Inst. of Physics IHEP.
Usability Issues Documentation J. Apostolakis for Geant4 16 January 2009.
EMI INFSO-RI SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader EMI First EC Review 22 June 2011, Brussels.
A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 Software Process panel SPI GRIDPP 7 th Collaboration Meeting 30 June – 2 July 2003 A.Aimar -
Maria Grazia Pia, INFN Genova Test & Analysis Project aka “statistical testing” Maria Grazia Pia, INFN Genova on behalf of the T&A team
Provide tools for the statistical comparison of distributions  equivalent reference distributions  experimental measurements  data from reference sources.
Alberto Ribon, CERN Statistical Testing Project Alberto Ribon, CERN on behalf of the Statistical Testing Team CLHEP Workshop CERN, 28 January 2003.
Maria Grazia Pia, INFN Genova Statistical Toolkit Recent updates M.G. Pia B. Mascialino, A. Pfeiffer, M.G. Pia, A. Ribon, P. Viarengo
Susanna Guatelli & Barbara Mascialino G.A.P. Cirrone (INFN LNS), G. Cuttone (INFN LNS), S. Donadio (INFN,Genova), S. Guatelli (INFN Genova), M. Maire (LAPP),
Geant4 Space User Workshop 2004 Maria Grazia Pia, INFN Genova Proposal of a Space Radiation Environment Generator interfaced to Geant4 S. Guatelli 1, P.
An update on the Statistical Toolkit Barbara Mascialino, Maria Grazia Pia, Andreas Pfeiffer, Alberto Ribon, Paolo Viarengo July 19 th, 2005.
SEAL Core Libraries and Services CLHEP Workshop 28 January 2003 P. Mato / CERN Shared Environment for Applications at LHC.
Maria Grazia Pia, INFN Genova Update on the Goodness of Fit Toolkit M.G. Pia B. Mascialino, A. Pfeiffer, M.G. Pia, A. Ribon, P. Viarengo
Business Analysis. Business Analysis Concepts Enterprise Analysis ► Identify business opportunities ► Understand the business strategy ► Identify Business.
Firmware - 1 CMS Upgrade Workshop October SLHC CMS Firmware SLHC CMS Firmware Organization, Validation, and Commissioning M. Schulte, University.
Mantid Stakeholder Review Nick Draper 01/11/2007.
Geant4 Training 2006 Short Course Katsuya Amako (KEK) Gabriele Cosmo (CERN) Susanna Guatelli (INFN Genova) Aatos Heikkinen (Helsinki Institute of Physics)
Maria Grazia Pia, INFN Genova Statistics Toolkit Project Maria Grazia Pia, INFN Genova AIDA Workshop.
The Statistical Testing Project Stefania Donadio and Barbara Mascialino January 15 TH, 2003.
Overview of RUP Lunch and Learn. Overview of RUP © 2008 Cardinal Solutions Group 2 Welcome  Introductions  What is your experience with RUP  What is.
G.Govi CERN/IT-DB 1 September 26, 2003 POOL Integration, Testing and Release Procedure Integration  Packages structure  External dependencies  Configuration.
SEAL Project Overview LCG-AA Internal Review October 2003 P. Mato / CERN.
23/2/2000Status of GAUDI 1 P. Mato / CERN Computing meeting, LHCb Week 23 February 2000.
Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Monte Carlo Chattanooga, April 2005 B. Mascialino, A. Pfeiffer, M. G. Pia, A. Ribon,
A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Software Process & Infrastructure for LCG Project Overview LCG Application Area Internal.
P. Nieminen & M.G. Pia Summary of the LowE e.m. WG week and common LowE/Standard e.m. sessions Petteri Nieminen & Maria Grazia Pia Geant4 Workshop Paris,
Paul Alexander 2 nd SKADS Workshop October 2007 SKA and SKADS Costing The Future Paul Alexander Andrew Faulkner, Rosie Bolton.
ANALYSIS PHASE OF BUSINESS SYSTEM DEVELOPMENT METHODOLOGY.
12 March, 2002 LCG Applications Area - Introduction slide 1 LCG Applications Session LCG Launch Workshop March 12, 2002 John Harvey, CERN LHCb Computing.
INFSO-RI SA2 ETICS2 first Review Valerio Venturi INFN Bruxelles, 3 April 2009 Infrastructure Support.
Maria Grazia Pia, INFN Genova and CERN1 Geant4 highlights of relevance for medical physics applications Maria Grazia Pia INFN Genova and CERN.
Maria Grazia Pia, INFN Genova - G4 WG Coord. Meeting, 13/11/2001 ow Energy Electromagnetic Physics ow Energy Electromagnetic Physics New physics features.
Follow-up to SFT Review (2009/2010) Priorities and Organization for 2011 and 2012.
CMS Experience with the Common Analysis Framework I. Fisk & M. Girone Experience in CMS with the Common Analysis Framework Ian Fisk & Maria Girone 1.
TK2023 Object-Oriented Software Engineering
Lecture 3 Prescriptive Process Models
Advanced Examples Maria Grazia Pia, INFN Genova
Update on the Goodness of Fit Toolkit
Unified Process Source & Courtesy: Jing Zou.
Advanced examples Test & Analysis Project LowE e.m. physics
A Statistical Toolkit for Data Analysis
Advanced examples Test & Analysis Project LowE e.m. physics
Data analysis in HEP: a statistical toolkit
B.Mascialino, A.Pfeiffer, M.G.Pia, A.Ribon, P.Viarengo
G.Cosmo - URD improvements Gabriele Cosmo (CERN/IT-API)
Short Course Siena, 5-6 October 2006
An update on the Goodness of Fit Statistical Toolkit
Statistical Testing Project
Comparison of data distributions: the power of Goodness-of-Fit Tests
Data analysis in HEP: a statistical toolkit
Presentation transcript:

Maria Grazia Pia, INFN Genova Statistical Testing Project Maria Grazia Pia, INFN Genova on behalf of the Statistical Testing Team LCG-Application Meeting CERN, 27 November 2002

Maria Grazia Pia, INFN Genova History and background

Maria Grazia Pia, INFN Genova What is? Provide tools for the statistical comparison of distributions – equivalent reference distributions (for instance, regression testing) – experimental measurements – data from reference sources – functions deriving from theoretical calculations or from fits physics validation regression testing system testing Main application areas in Geant4: Interest in other areas, not only Geant4? LCG? A project to develop a statistical analysis system, to be used in Geant4 testing A project to develop a statistical analysis system, to be used in Geant4 testing

Maria Grazia Pia, INFN Genova History Statistical testing agreed in the Geant4 Collaboration as a major objective for 2002 Initial ideas presented at Geant4 TSB meeting, November 2001 Open brainstorming session at a Geant4-WG workshop, 31 May 2002 Inception phase, summer 2002 –Informal discussions with STT, Geant4 collaborators and interested potential developers –Initial collection of user requirements in Geant4 –First version of software process deliverables: Vision, URD, Risk List Presentation at Geant4 Workshop + parallel sessions, October 2002 – Launch of the project

Maria Grazia Pia, INFN Genova The team Development team Pablo Cirrone, INFN Southern National Lab Stefania Donadio, Univ. and INFN Genova Susanna Guatelli, CERN/IT/API Technical Student and INFN Genova Alberto Lemut, Univ. and INFN Genova Barbara Mascialino, Univ. and INFN Genova Sandra Parlati, INFN Gran Sasso National Lab Andreas Pfeiffer, CERN/IT/API Maria Grazia Pia, INFN Genova Geant4 system integration team Gabriele Cosmo, CERN/IT/API - Geant4 Release Manager Sergei Sadilov, CERN/IT/API - Geant4 System Testing Coordinator Statistical consultancy Paolo Viarengo, Univ. Genova, Statistician interested collaborators are welcome! + requirements, suggestions, -testing by many other Geant4 Collaborators (M. Maire, A. Ribon, L. Urban et al.)

Maria Grazia Pia, INFN Genova The vision

Maria Grazia Pia, INFN Genova Vision: the basics software process Rigorous software process vision Have a vision for the project –An internal tool for Geant4 physics & STT? –Also for Geant4 physics validation in the experiments? –Other parties than Geant4 interested? stakeholders Who are the stakeholders? users Who are the users? developers Who are the developers? architecture Build on a solid architecture Clearly define scopeobjectives scope, objectives Flexible, extensible, maintainable Flexible, extensible, maintainable system quality Software quality Clearly define roles

Maria Grazia Pia, INFN Genova Scope of the project tools for statistical testing The project will provide tools for statistical testing of Geant4 –physics comparisons and regression testing –multiple comparison algorithms Generality Generality (for application also in other areas) should be pursued –facilitated by a component-based architecture The statistical tools should be used in Geant4 (and in other frameworks) –tool to be used in testing frameworks –not a testing framework itself Re-use existing tools whenever possible –no attempt to re-invent the wheel –but critical, scientific evaluation of candidate tools

Maria Grazia Pia, INFN Genova Architectural guidelines architectural The project adopts a solid architectural approach functionalityquality –to offer the functionality and the quality needed by the users maintainable –to be maintainable over a large time scale extensible –to be extensible, to accommodate future evolutions of the requirements Component-based approach –Geant4-specific + general –Geant4-specific components + general components –to facilitate re-use and integration in diverse frameworksAIDA –adopt a (HEP) standard –no dependence on any specific analysis toolPython LCG Architecture Blueprint RTAG The approach adopted is compatible with the recommendations of the LCG Architecture Blueprint RTAG

Maria Grazia Pia, INFN Genova The reason why we are here… statistics comparison component Core statistics comparison component + user layer can be generalised to wider scope than Geant4 only This is the reason why we present the project to LCG scientific discussion –to establish a scientific discussion on a topic of common interest interested users –to see if there are any interested users interested collaborators –to see if there are any interested collaborators We would all benefit of a collaborative approach to a common problem –share expertise, ideas, tools, resources…

Maria Grazia Pia, INFN Genova Software process guidelines Significant experience in the team –in Geant4 and in other projects ISO Guidance from ISO –standard! USDPtailored USDP, specifically tailored to the project RUP –practical guidance and tools from the RUP –both rigorous and lightweight –mapping onto ISO LCG Software Process Infrastructure Open to use tools provided by the LCG Software Process Infrastructure project

Maria Grazia Pia, INFN Genova Who are the stakeholders? NameDescriptionResponsibilities Geant4 STT Coordinator Coordinates system testing Ensure that the system meets the needs of Geant4 System Testing Geant4 physics coordinators Coordinate Geant4 std EM, lowE EM, hadronic WGs Ensure that the system meets the needs of Geant4 Physics Testing Geant4 TSB Is responsible for Geant4 technical matters Provide guidelines, monitors progress INFN Computing Committee National Committee whom part of the developers respond to; has appointed 4 referees Recommend funding; review the project, monitor progress Others?Who? LCG?Requirements? Expertise?

Maria Grazia Pia, INFN Genova Who are the users? Other potential users: users of the Geant4 Toolkit users of the Geant4 Toolkit, wishing to compare the results of their applications to reference data or to their own experimental results other projects other projects with requirements for statistical comparisons of distributions (e.g. the LHC Computing Grid project) GroupsResponsibilities Geant4 physics Working Groups Provide and document requirements, provide feedback on prototypes, perform -testing on preliminary releases of the product, provide use cases for acceptance testing Geant4 STT Provide and document requirements, perform formal acceptance testing for adoption in system testing

Maria Grazia Pia, INFN Genova Some use cases Regression testing –Throughout the software life-cycle Online DAQ –Monitoring detector behaviour w.r.t. a reference Simulation validation –Comparison with experimental data Reconstruction –Comparison of reconstructed vs. expected distributions Physics analysis –Comparisons of experimental distributions (ATLAS vs. CMS Higgs?) –Comparison with theoretical distributions (data vs. Standard Model)

Maria Grazia Pia, INFN Genova What do the users want? User requirementsGeant4 User requirements from Geant4 (physics, system testing) elicited, analysed, specified and reviewed with the users –User Requirements Document – –Use case model in progress statisticalcomponent Specific user requirements related to the core statistical component –Detail in progress (URD in preparation) –Input from LCG? Requirement traceability –Analysis/design, implementation, test, documentation, results

Maria Grazia Pia, INFN Genova Are there any constraints? Geant4 constraint requirements Based on AIDA No concrete dependencies on specific AIDA implementations should appear in the code of the system tests Available on Geant4 supported platforms The system should not require additional licenses w.r.t. what required for Geant4 development Other non-functional requirements?

Maria Grazia Pia, INFN Genova The core statistical component

Maria Grazia Pia, INFN Genova HBOOK, PAW & Co. the most powerful Based on considerations such as those given above, as well as considerable computational experience, it is generally believed that tests like the Kolmogorov or Smirnov-Cramer-Von-Mises (which is similar but more complicated to calculate) are probably the most powerful for the kinds of phenomena generally of interest to high-energy physicists. […] provided the data are not binned. The value of PROB returned by HDIFF is calculated such that it will be uniformly distributed between zero and one for compatible histograms, provided the data are not binned. […] notcorrect binned data The value of PROB should not be expected to have exactly the correct distribution for binned data. HBOOK manual, 1994 CDF Collaboration, Inclusive jet cross section in p pbar collisions at sqrt(s) 1.8 TeV, Phys. Rev. Lett. 77 (1996) 438 but…

Maria Grazia Pia, INFN Genova Goodness-of-fit tests Pearsons 2 test Kolmogorov test Kolmogorov – Smirnov test Lilliefors test Cramer-von Mises test Anderson-Darling test Kuiper test … It is a difficult domain… Implementing algorithms is easy But comparing real-life distributions is not easy Incremental and iterative software process Collaboration with statistics experts Patience, humility, time… System open to extension and evolution Suggestions welcome!

Maria Grazia Pia, INFN Genova Pearsons 2 discrete Applies to discrete distributions It can be useful also in case of continuous distributions, but the data must be grouped into classes Cannot be applied if the counting of the theoretical frequencies in each class is < 5 When this is not the case, one could try to unify contiguous classes until the minimum theoretical frequency is reached

Maria Grazia Pia, INFN Genova Kolmogorov test The easiest among non-parametric tests continuous Verify the adaptation of a sample coming from a random continuous variable Based on the computation of the maximum distance between an empirical repartition function and the theoretical repartition one Test statistics: D = sup | F O (x) - F T (x)|

Maria Grazia Pia, INFN Genova Kolmogorov-Smirnov test Problem of the two samples –mathematically similar to Kolmogorovs Instead of comparing an empirical distribution with a theoretical one, try to find the maximum difference between the distributions of the two samples F n and G m : D mn = sup |F n (x) - G m (x)| continuous Can be applied only to continuous random variables Conover (1971) and Gibbons and Chakraborti (1992) tried to extend it to cases of discrete random variables

Maria Grazia Pia, INFN Genova Lilliefors test Similar to Kolmogorov test Based on the null hypothesis that the random continuous variable is normally distributed N(m, 2 ), with m and 2 unknown Performed comparing the empirical repartition function F(z 1,z 2,...,z n ) with the one of the standardized normal distribution (z): D* = sup | F O (z) - (z)|

Maria Grazia Pia, INFN Genova Cramer-von Mises test Based on the test statistics: 2 = integral (F O (x) - F T (x)) 2 dF(x) continuousdiscrete Can be performed both on continuous and discrete variables Satisfactory for symmetric and right-skewed distributions

Maria Grazia Pia, INFN Genova Anderson-Darling test Performed on the test statistics: A 2 = integral { [F O (x) – F T (x)] 2 / [F T (x) (1-F T (X))] } dF T (x) continuousdiscrete Can be performed both on continuous and discrete variables skewness Seems to be suitable to any data-set (Aksenov and Savageau ) with any skewness (symmetric distributions, left or right skewed) Seems to be sensitive to fat tail of distributions

Maria Grazia Pia, INFN Genova Kuiper test Based on a quantity that remains invariant for any shift or re-parameterization Does not work well on tails D* = max (F O (x)-F T (x)) + max (F T (x)-F O (x))

Maria Grazia Pia, INFN Genova Work in progress

Maria Grazia Pia, INFN Genova OOAD in progress Preliminary design of the statistical component in progress Core statistics comparison package User layer Policy-based class design Validation of the design through use cases Some open issues identified, to be addressed in next design iteration

Maria Grazia Pia, INFN Genova work in progress + more algorithms

Maria Grazia Pia, INFN Genova work in progress

Maria Grazia Pia, INFN Genova work in progress Use case: compare two continuous distributions

Maria Grazia Pia, INFN Genova Work in progress Implementation and test of preliminary design What can be re-used? –Algorithms in GSL, NAG libraries (to be evaluated) Studies in progress –Transformation between continuous-discrete distributions –Strategies to use Kolmogorov-Smirnov with discrete distributions (E. Dagum + original ideas) –How to deal with experimental errors (not only statistical!) –Multi-dimensional distributions –Bayesian approach In the to-do list –Conversion from AIDA objects to distributions –Pythonisation Revision of the initial documents (Vision, URD, Risks) –Based on the recent evolutions in the project –Input from todays meeting?

Maria Grazia Pia, INFN Genova Work in progress: Geant4-specific Development of general physics tests in the E.M. domain, for comparison of reference distributions –Compilation of existing tests –Evaluation, documentation of tests –Elicitation of requirements for tests among the Geant4 physics groups –Collection of reference data/distributions Prototype for automated comparison w.r.t. reference databases –NIST, Sandia etc., directly downloaded from the web –Prototype as a risk mitigation strategy Integration in the Geant4 system testing framework Integration in Geant4 physics testing frameworks

Maria Grazia Pia, INFN Genova Where? Geant4-specific stuff –In Geant4 –May be included in public distribution, if of interest to users Core statistical component –Developed in an independent CVS repository –Code, documentation, software process deliverables Web site – Contact persons

Maria Grazia Pia, INFN Genova Time scale driven by Geant4 needs Aggressive time scale driven by Geant4 needs –incremental and iterative software process OOAD + implementation already started Prototype at CHEP Advanced functional system summer 2003 Open to the needs/suggestions of LCG –compatible with the available resources and Geant4 needs

Maria Grazia Pia, INFN Genova Conclusions… Geant4 requires a statistical testing system for physics validation and regression testing –to provide a high quality product to its user communities Core statistical component (of potential general interest) Geant4-specific components Project compatible with LCG architecture blueprint –component-based approach, AIDA, Python… Rigorous software process –to contribute to the quality of the product Aggressive time scale dictated by Geant4 needs Open to scientific collaboration Beginning…