1. Maria Grazia Pia, INFN Genova Methods and techniques for Monte Carlo physics validation MC 2015 19-23 April 2015, Nashville, TN, USA C. Choi, M. C. Han, S. Hauf, G. Hoff, C. H. Kim, H. S. Kim, S. H. Kim, M. Kuster, M. G. Pia, P. Saracco, G. Weidenspointner Hanyang University, Seoul, Korea EU XFEL GmbH, Hamburg, Germany CAPES, Brasilia, Brazil MPE, Garching, Germany Foreword Due to limited time allocation, there is room to highlight concepts only Details are documented and discussed in dedicated journal publications

2. Maria Grazia Pia, INFN Genova In the literature… Limited documentation of simulation validation ‒ Mostly in the form of specific use cases compared to measurements in the same experimental scenario ▻ Do they apply to similar/different use cases? ▻ How to extrapolate the results to different scenarios? Hardly any validation of the basic physics models implemented in Monte Carlo codes ‒ Why? Ongoing projects on uncertainty quantification ‒ Methods to predict the uncertainty of simulation observables based on knowledge of the uncertainties of simulation “ingredients” (quantitative) 2

3. Maria Grazia Pia, INFN Genova What is what Verification Validation Calibration 3 IEEE Standard 1012 Conforms to  ISO/IEC 15288 (IEEE Std 15288) Systems and Software Engineering – System Life Cycle Processes  ISO/IEC 12207 (IEEE Std 12207) Systems and Software Engineering – Software Life Cycle Processes  IEEE Std 1074 IEEE Standard for Developing a Software Project Life Cycle Process

4. Maria Grazia Pia, INFN Genova Verification A.The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. B.The process of providing objective evidence that the system, software, or hardware and its associated products conform to requirements (e.g., for correctness, completeness, consistency, and accuracy) for all life cycle activities during each life cycle process (acquisition, supply, development, operation, and maintenance); satisfy standards, practices, and conventions during life cycle processes; and successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities. 4 e.g. in the context of Monte Carlo simulation Requirement: Compton scattering cross section shall be described by the Klein-Nishina formula Verification: the software calculates consistently, correctly, with adequate numerical precision…

5. Maria Grazia Pia, INFN Genova Validation A.The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements. B.The process of providing evidence that the system, software, or hardware and its associated products satisfy requirements allocated to it at the end of each life cycle activity, solve the right problem (e.g., correctly model physical laws, implement business rules, and use the proper system assumptions), and satisfy intended use and user needs. 5 In the context of Monte Carlo simulation validation consistency with experimental measurements e.g. does the Klein-Nishina formula reproduce measured differential cross sections of photon inelastic scattering? e.g. does the Klein-Nishina formula reproduce measured differential cross sections of photon inelastic scattering?

6. Maria Grazia Pia, INFN Genova Calibration The process of improving the agreement of a code calculation with respect to a chosen set of benchmarks through the adjustment of parameters implemented in the code Calibration is not validation ‒ Validation is the process of confirming that the predictions of a code adequately represent measured physical phenomena 6 T. G. Trucano et al., Calibration, validation, and sensitivity analysis: What's what, Reliability Eng. & System Safety, vol. 91, no. 10-11, pp. 1331-1357, 2006 M. G. Pia et al, Physics-related epistemic uncertainties of proton depth dose simulation, IEEE Trans. Nucl. Sci., vol. 57, no. 5, pp. 2805-2830, 2010 AKA “tuning”

7. Maria Grazia Pia, INFN Genova What is NOT validation Comparison of simulations using different Monte Carlo codes ‒ Or comparison of different simulation models Comparison of simulation with theory ‒ Or so-called “analytical calculations” Comparison of simulation with non-pertinent experimental data Calibration Oenology Mozart opera 7

8. Maria Grazia Pia, INFN Genova Establishing validity Comparison of simulation results and experimental data in the literature mainly rests on qualitative visual appraisal of figures indicators (%) deprived of any statistical meaning Agreement Good agreement Excellent agreement Satisfactory agreement … 8

9. Maria Grazia Pia, INFN Genova Statistics Mathematical foundation of Monte Carlo physics validation Rigorous statistical methods assess ‒ Whether a simulation model is consistent with nature ▻ well, whether a simulation model is not inconsistent with nature… ‒ Whether different simulation models produce (or do not produce) equivalent results in terms of compatibility with experiment Hypothesis testing ‒ Well established methods 9   2  Kolmogorov-Smirnov  Anderson-Darling  Cramer-von Mises  etc.   2  Kolmogorov-Smirnov  Anderson-Darling  Cramer-von Mises  etc.  Fisher exact text  Barnard test   2  etc.  Fisher exact text  Barnard test   2  etc. Goodness-of-fit tests Categorical data analysis Mainly applied to contingency tables

10. Maria Grazia Pia, INFN Genova What is validated Validation of the “ingredients” of Monte Carlo codes ‒ The foundation of physics models used in the code ‒ Cross sections (total, partial, differential) ‒ Secondary particle production ‒ Atomic and nuclear parameters (e.g. binding energies, transition probabilities etc.) Validation of simulated observables produced by Monte Carlo codes in use cases ‒ Largely represented in the literature ‒ Often qualitative only ‒ Seldom related to “physics ingredients” 10

11. Maria Grazia Pia, INFN Genova How is validation performed? Validation of basic physics “ingredients” Unit tests Validation of simulated observables Simulation applications Testability must be embedded in the software design to enable physics unit tests Amending the software design of a mature Monte Carlo system that did not account for testability is expensive 11 complementary

12. Maria Grazia Pia, INFN Genova Post-RD44 Geant4 electromagnetic software design 12 One needs a geometry (and a full scale application) to test any photon cross section Difficult to test  no testing often Reverse engineered G4VEmProcess G4VEnergyLossProcess G4VMultipleScattering G4VEmModel Attributes abstract class Operations

13. Maria Grazia Pia, INFN Genova Discipline of software engineering Most of the problems with physics tests can be easily solved if we simply write tests as we develop our code ‒ …and we maintain the tests ‒ …and we regularly execute them ‒ …and we investigate the reasons for failure Software design reviews: care about testability 13 If a test is hard to write, that means that we have to find a different design which is testable

14. Maria Grazia Pia, INFN Genova Ongoing activity Extensive R&D on Geant4 physics validation Software design ‒ Enables testability ‒ Facilitates the validation of a wide set of modeling options, including some that have not yet been used in Monte Carlo codes Validation of basic physics models and parameters ‒ Electron-photon interaction cross sections, atomic binding energies, radiative transition probabilities etc. Validation of simple observables of general interest ‒ Recent project on electron backscattering validation Uncertainty quantification ‒ Original method, further R&D in progress 14

15. Maria Grazia Pia, INFN Genova Detangling Testable Open - closed Photoionisation New models Handles any tabulated cross section Can be validated in a unit test Cross section models can be compared with statistical categorical tests 15

16. Maria Grazia Pia, INFN Genova Tools for statistical analysis 16 The Statistical Toolkit Large collection of algorithms for goodness-of-fit testing Two-sample problem: comparing two distributions aidaR Bridge between iAIDA and R Our team developed software tools for statistical data analysis specifically to support simulation validation

17. Maria Grazia Pia, INFN Genova A sample of validation results 17 More extensive information in journal publications Physics processes: photon interaction cross sections An example of complex observable: electron backscattering fraction

18. Maria Grazia Pia, INFN Genova Photoionisation cross section sources 18 YearCompilationEnergyZ(sub)ShellMethod 1967-1988 Biggs-Lighthill 10 eV – 100 GeV1-100-parameterised 1992 Brennan-Cowan 30 eV – 700 keV3-92-tabulated 2000 Chantler 10 eV – 433 keV1-92Ktabulated 2003 Ebel 1 keV – 300 keV1-92allparameterised 2002 Elam 100 eV – 1 MeV1-98-tabulated 1997 EPDL97 (Scofield) 10 eV – 100 GeV1-100alltabulated 1982-1993 Henke 10 eV – 30 keV1-92-tabulated 1970-2006 McMaster/Shaltout 1 keV – 700 keV1-94-tabulated 1989 PHOTX (Scofield) 1 keV – 100 MeV1-100tabulated 2001 RTAB 10 eV – 30 keV1-99alltabulated 1973 Scofield 1 keV – 1.5 MeV1-100alltabulated 1970 Storm-Israel 1 keV – 100 GeV1-100-tabulated 1973 Veigele 100 eV – 100 MeV1-94-tabulated 1987-2010 XCOM (Scofield) 1 keV – 100 GeV1-100-tabulated e.g. Chantler’s exchange potential in his DHF calculation is different from Scofield’s Different methods and calculations

19. Maria Grazia Pia, INFN Genova Total photoionisation cross sections Most calculation methods exhibit similar compatibility with experiment for E >250 eV ‒ Chantler, Brennan-Cowan look worse Degraded accuracy below 250 eV 19 preliminary Analysis of contingency tables EPDL Chantler EPDL Brennan-Cowan Fisher0.0440.011 Pearson  2 0.0330.007 Barnard0.0350.007 H O Fe

20. Maria Grazia Pia, INFN Genova Shell cross sections 20 shellEPDLChantlerRTABscRTABEbel K0.2090.350<0.0010.315<0.001 L10.075<0.0010.0690.964 L20.339<0.0010.2990.154 L31<0.00111 M1<0.001 M40.031<0.001 M5<0.001 N1<0.001 N6<0.001 N7<0.001 O1<0.001 O2<0.001 O3<0.001 P1<0.001 p-value  2 test Systematic effect observed with RTAB shell cross sections (presumably a missing factor in the calculation) Calculated inner shell cross sections compatible with experiment Outer shell cross sections inconsistent with experimental data Beware: small data sample, limited experimental sources Outer shell cross sections inconsistent with experimental data Beware: small data sample, limited experimental sources K L3L3 M4M4 O1O1

21. Maria Grazia Pia, INFN Genova Angular distribution 21 Qualitative appraisal Limited experimental sample Experimental systematic effects (corrected/uncorrected data) Qualitative appraisal Limited experimental sample Experimental systematic effects (corrected/uncorrected data) Option à la GEANT 3 (Sauter) evaluated along with other Geant4 options

22. Maria Grazia Pia, INFN Genova Photon elastic scattering Penelope EPDLRelativ.Non-Rel.ModifiedMFFRFFSM 20012008FF ASF NT  0.270.38 0.250.350.490.520.480.77 error±0.05±0.06 ±0.05±0.06 ±0.05 Form factor approximation: non relativistic, relativistic, modified + anomalous scattering factors 2 nd order S-matrix calculations recent calculations, not yet used in Monte Carlo codes  = fraction of test cases compatible with experiment, 0.01 significance Differential cross sections Statistical analysis, GoF + categorical

23. Maria Grazia Pia, INFN Genova Differential Compton scattering cross section modelefficiencyerror EPDL0.820.02 Penelope0.820.02 Klein-Nishina0.540.03 Brusa0.840.02 BrusaF0.840.02 PenBrusa0.840.02 PenBrusaF0.840.02 Biggs0.840.02 BiggsF0.850.02 Hubbell0.820.02 Work in progress! Various scattering functions are evaluated w.r.t. experimental data >2300 experimental data Geant4 standard Geant4 lowenergy

24. Maria Grazia Pia, INFN Genova e + e - pair production Total cross section: Bethe-Heitler with corrections (Hubbell, Gimm, Overbo) Near threshold 24 Geant4 standardEPDLXCOM p-value<0.0010.982<0.001 E>1.119 MeV Validation at high energy in progress

25. Maria Grazia Pia, INFN Genova Electron backscattering 25 Goudsmit-Saunderson Urban WentzelVI Single Coulomb scattering + Various Geant4 PhysicsLists with various configuration options Interplay of geometry and physics Urban model, “DistanceToBoundary step limitation option S. H. Kim, M. G. Pia, T. Basaglia, M. C. Han, G. Hoff, C. H. Kim, P. Saracco, Validation Test of Geant4 Simulation of Electron Backscattering, IEEE Trans. Nucl. Sci., vol. 62, no. 2, pp. 451-479, Apr. 2015. T. Basaglia, M. C. Han, G. Hoff, C. H. Kim, S. H. Kim, M. G. Pia, P. Saracco, Investigation of Geant4 Simulation of Electron Backscattering, IEEE Trans. Nucl. Sci., submitted March 2015. Further work in progress

26. Maria Grazia Pia, INFN Genova Uncertainty quantification 26 Input observable with uncertainties Monte Carlo method Statistical uncertainty Uncertainty quantification is the ground for predictive Monte Carlo simulation Beware: input uncertainties can be hidden in the code (in models and algorithms) Validation of MC modeling ingredients Parameter uncertainties N18-5 Progress with Uncertainty Quantification in Generic Monte Carlo Simulations cross sections, branching ratios, physics models, physics parameters..

27. Maria Grazia Pia, INFN Genova Conclusion Detector design, experimental strategies, physics results depend critically on software Monte Carlo simulation plays a crucial role in many experimental domains Methods of simulation validation ‒ Basic physics and complex experimental observables ‒ Unit tests and full simulation applications ‒ Quantification through statistical methods Testability embedded in the software design ‒ Since the early stages of the software development Ongoing effort to make Geant4 physics testable http://www.ge.infn.it/geant4/papers and to test it