Software reliability with GAMOS P. Arce1, A. Dotti2 1 CIEMAT, Medical Applications Unit, Madrid, Spain 2 SLAC National Accelerator Laboratory, California, USA Nuclear Science Symposium and Medical Imaging Conference, Atlanta, Georgia, USA 21-28 October 2017

Outline Controlling the verbosity CPU optimization What this talk is about Introduction to GAMOS MC applications An easy and flexible framework Checking the physics Precise visualization Obtaining detailed data GAMOS data Filtering Classifying Scoring Controlling the verbosity CPU optimization Where time is spent Physics cuts optimization General importance sampling Other optimization techniques Testing the code robustness Summary Pedro Arce Latest Developments in GAMOS

What this talk is about I am not going to talk about how much GAMOS is reliable the quality assurance tests the dozens of scientific publications the thousands of users the ten-year experience of the framework (plus ten years more doing similar Geant4-based frameworks) … But about the tools that GAMOS offers you to make your own Monte Carlo simulation more reliable Pedro Arce Latest Developments in GAMOS

Monte Carlo applications Often the use a MC simulation is a difficult task for a non-expert software user In the case of Geant4 most of the application has to be written in C++ Several applications try to facilitate the use of MC in a specific field Providing an scripting language tailored to the field Pedro Arce

Monte Carlo applications But users find some problems with these applications User wants to describe some input not included in the application A peculiar volume shape, a new primary generator position distribution, some option in the physics,… User wants to have some kind of output or detailed information to debug or understand some part of the simulation, and the application only provides a limited amount of output possibilities Dose from the gammas that entered the phantom with small energy, energy lost by particle traversing a volume as a function of the initial energy, … The available applications cover a limited amount of physics fields, and many users do not find an appropriate one for their needs Pedro Arce

/e- radiotherapy, proton/ion radiotherapy, An easy… An scripting language, instead of C++, plus many tools to facilitate the definition of input and output Any geometry in a text file format Including superposition's of parallel geometries Several modules to define in a few lines the most complicated parts (jaws, multi-leaf collimators, range modulators,…) Dozens of distributions for primary particles: position, direction, energy and time Any available Geant4 physics + 30 scorers, including error calculation Many optimization options F18 decay energy Many applications focused of a physics field: /e- radiotherapy, proton/ion radiotherapy, PET, SPECT, Compton Camera, tissue optics,  spectroscopy, shielding Pedro Arce

… and flexible framework Extensive use of plug-in technology User can easily extend the framework to satisfy a new requirement Any Geant4 example can be transformed into a GAMOS example Not behaving as a black box, but letting the user understand in detail each aspect of the simulation A new concept, GAMOS data, plus the use of filters and classifiers, allows to satisfy with a few user commands requirements as complex as: Write in a file the logarithm of the energy of the gammas that reach the phantom only if they have left some energy in the jaws Plot the X vs. Y position of each step in the source volume only if the particle or one of its descendants will reach the detector Flexible use of the verbosity of each event/track /track step and each package independently Pedro Arce

e- bremsstrahlung models Checking the physics Livermore physics Penelope physics There is a wide variety of physics options in Geant4 Alternative processes, models and parameters User should to check different options to guarantee the best match to the experimental data To help in this task, GAMOS offers the possibility practically all the physics options available in Geant4 with a few user commands Standard, Livermore, Penelope electromagnetic physics models + switch on/off: decay, radioactiveDecay, opticalphoton processes, Cerenkov, gamma-nuclear, electron-nuclear, positron-nuclear, xray-refraction (new model in GAMOS), gamma polarization, atomic deexcitation Choose multiple scattering model: Urban, Wentzel-VI, Goudsmit-Saunderson, single scattering Choose bremsstrahlung model: Tsai, Koch-Motz 2BN, Koch-Motz 2BS, dipBust Nanodosimetry physics: DNA and LEPTS models, Alll Geant4 physics list for hadronic models e- bremsstrahlung models Plus an easy way to use your own physics list in GAMOS scripts Add one line: PLUGINSVC_FACTORY(ExUCNPhysicsList,G4VUserPhysicsList*()) And use it in the GAMOS script: /gamos/physics ExUCNPhysicsList Pedro Arce 8 8 8

Placement in wrong mother Wrong pinhole dimensions detected Precise visualization Visualization is often needed to check the correctness of the geometry simulation But sometimes 3D visualization is not the best option GAMOS allows visualization with any Geant4 drive (although remember that some of them require the installation of an external package) And also provides a 2D visualization Based on ray tracing (real tracking) 3D VRML view: everything looks OK 2D GAMOS view: Placement in wrong mother detected 2D GAMOS view: Wrong pinhole dimensions detected (should be 2, not 3 mm) 3D VRML view: everything looks OK Pedro Arce

Obtaining detailed data Most users are researchers, it is not enough to provide some final results, like dose distribution, or PET event classification table Want to have a deep understanding of what happens in the simulation Want to have the capability to evaluate the reliability of the results Want to choose the best physics configuration For example: How many gammas traverse completely the jaws? How much energy they lose? What is the length travelled by electrons produced by Compton interactions in a crystal? Dump in a binary file the position of the gammas as they cross a human body only if in the future this track or one of its descendants will leave a signal in one detector Make an histogram of the energy deposited in a water volume by any of the electrons that were created in a Compton interaction in the jaws volumes, only if they have an energy bigger than 1 MeV when they enter the water volume … Something we GAMOS developers have never imagined… Pedro Arce

GAMOS data A new concept developed in GAMOS Almost 200 types of data can be obtained at each step, track, event or run event ID, trackID, particle, process, kinetic energy, energy lost, energy deposited, step length, number of secondary's, position (X/Y/Z/R2/R3/phi/theta), direction, change in position, change in angle, energy secondary/energy primary, angle secondary – primary, etc. Also composed data, using arithmetic expressions sqrt(2.*FinalLocalPosX*FinalLocalPosY) With a few user commands data can be used to fill histograms (1D, 2D, profile), be dumped in text or binary files, or be printed on the screen EXAMPLE: Plot at each track step the logarithm of the initial kinetic energy and also the final X position vs final Y position /gamos/userAction GmStepDataHistosUA /gamos/setParam GmStepDataHistosUA:DataList log10(InitialKineticEnergy) FinalPosX.vs.FinalPosY Pedro Arce

A filter is an algorithm that accepts or reject a step or track Filtering A filter is an algorithm that accepts or reject a step or track ● > 100 filters in GAMOS Starts/Ends/Enters/Exits/Traverses/In LogicalVolume/PhysicalVolume/Touchable/Region, or any of their children, particle type, process type, primary or secondary, charged or neutral, etc. Composed filters: ● AND, OR, XOR, Inverse, OnSecondary (apply filter to secondary tracks created in a step), with history (pass filter in any previous track step or any ancestor previous track step), etc. Filter on numeric data: if the data is inside an interval /gamos/filter energyF GmNumericDataFilter InitialKineticEnergy 100.*keV 1.*MeV Filter on string data: if the data is included in a list /gamos/filter scatteringF GmStringDataFilter Process compt rayleigh Pedro Arce

Filtering example Dump in a text file the energy lost by the tracks in all kind of jaws # only when track steps happen in JAWS_X or JAWS_Y /gamos/filter jawsF GmInLogicalVolumeFilter MY_JAWS* # Select what to dump into text file /gamos/setParam GmTrackDataTextFileUA_jawsF:DataList EventID TrackID Particle AccumulatedEnergyLost # Include a header line in text file /gamos/setParam GmTrackDataTextFileUA_jawsF:WriteHeader 1 # Dump data in text file /gamos/userAction GmTrackDataTextFileUA jawsF HEADER: 4,"EventID","TrackID","Particle","AccumulatedEnergyLost" 2,2,"e-",0.0198374 40,2,"e-",0.151526 63,2,"e-",0.188402 119,52,"e-",0.494147 119,49,"gamma",0.514096 119,46,"gamma",4.02731 Pedro Arce

Classifying A new concept developed in GAMOS A classifier is an algorithm that classifies a step or track information, returning an index ● ByParticle, ByKineticEnergy, ByNAncestors, ByLogicalVolume, ByPhysicalVolume, ByTouchable, ByRegion, ByProcess, ByMaterial, etc. Compound classifiers: return an index combining two or more classifiers /gamos/classifier particleAndProcessC GmClassifierByParticle GmClassifierByProcess Classifier on numeric data: different index for each interval /gamos/classifier energyC GmClassifierByNumericData InitialKineticEnergy 0. 10.*MeV 1*MeV Classifier on string data: different index for each data value /gamos/classifier processC GmClassifierByStringData CreatorProcess Pedro Arce

Classifying examples Plot for each Z layer the X vs Y position only when an interaction occurred in the “phantom” volume # only when track steps happen in phantom /gamos/filter InPhantomF GmInLogicalVolumeFilter phantom # classify by the Z position (a different histogram for each Z interval) /gamos/classifier ZposC GmClassifierByNumericData FinalPosZ -100. 100. 200./10 # select what to plot /gamos/setParam GmStepDataHistosUA_InPhantomF_ZposC:DataList FinalPosX.vs.FinalPosY # build histograms at each step using filter and classifier gamos/userAction GmStepDataHistosUA InPhantomF ZposC -100 < Z < -80 -80 < Z < -60 -60 < Z < -40 Pedro Arce

Scoring ● Dose ● Deposited energy ● Flux (in/out/passage) Scoring may be an important part of a simulation  powerful and flexible framework developed, fully based on user commands: Many possible quantities can be scored in one or several volumes (based on Geant4 scorers) ● Dose ● Deposited energy ● Flux (in/out/passage) ● Current (in/out/passage) ● Charge ● Step length ● Number of particles ● Number of interactions ● Number of 2ary particles ● Number of steps ● Minimum kinetic energy ● Kerma For each scored quantity one of several filters can be used only electrons, only particles with energy in a given interval, … Several ways to classify the different scores One different score for each volume copy, or volume name, or energy bin, … Results can be printed in one or several formats for each scored quantity Standard output, text/binary file, histograms Scoring can be made in real or in parallel worlds All scored quantities can be calculated with/without errors All scored quantities can be calculated per event or per run Taking into account correlations from particles from same event Pedro Arce

Scoring Everything managed with user commands NFWOKX OUTPUT: For example: score the number of interactions (collisions) of gammas in each volume copy - Select all volumes for scoring /gamos/scoring/createMFDetector nInterDet * - Score number of interactions /gamos/scoring/addScorer2MFD nInterScorer GmG4PSNofCollision nInterDet - One score per each physical volume /gamos/scoring/assignClassifier2Scorer GmClassifierByPhysicalVolume nInterScorer - Only score for gammas /gamos/scoring/addFilter2Scorer GmGammaFilter nInterScorer OUTPUT: NFWOKX MultiFunctionalDet: nInterDet PrimitiveScorer: nInterScorer Number of entries= 6 index: target:1 = 2.111 +-(REL) 0.024593555 index: primary collimator:1 = 1.066 +-(REL) 0.049482843 index: flattening filter:1 = 0.028 +-(REL) 0.28856012 index: jaws_X:1 = 0.003 +-(REL) 1 index: jaws_X:2 = 0.006 +-(REL) 0.6231189 index: expHall:0 = 0.002 +-(REL) 0.70675279 Pedro Arce

Controlling the verbosity High verbosity is often necessary Find out the reason for a strange behavior Better understand some result But a too big verbosity output would hamper the task Verbosity should have a high degree of granularity Switch on the verbosity only for a selected group of events, or even tracks Control the degree of verbosity of each simulation field (geometry, particle generator, scoring, …) individually All this can be done in GAMOS with the extra advantage of keeping a full control through user commands ### GAMOS SENSITIVE DETECTOR VERBOSITY IN EVENT 1138 /gamos/userAction GmGamosVerboseByEventUA /gamos/verbosity/byEvent GmSDVerbosity debug 1138 1138 ### GEANT4 VERBOSITY IN EVENT 1138 /gamos/setParam GmTrackingVerboseUA:EventMin 1138 /gamos/setParam GmTrackingVerboseUA:EventMax 1138 /gamos/userAction GmTrackingVerboseUA Pedro Arce

CPU time Optimization Resources are limited: so CPU optimization means Smaller statistical error (often without optimization error is not satisfactory) More detailed checks can be done to guarantee the reliability of the results GAMOS offers many and diverse tools for CPU time optimization First: an utility to study where CPU time is spent, with user commands By particle, energy bins, volume, region, … (or combination of them) Just add user commands: /gamos/classifier ParticleAndEnergyClassifier GmCompoundClassifier GmClassifierByParticle GmClassifierByEnergy /gamos/userAction GmTimeStudyUA ParticleAndEnergyClassifier gamma/0.01-0.1: User=2.01 Real=2.45 Sys=0.27 gamma/0.1-1: User=19.12 Real=22.05 Sys=1.51 gamma/1-10: User=4.25 Real=5.4 Sys=0.3 e-/0.01-0.1: User=4.71 Real=5.41 Sys=0.38 e-/0.1-1: User=15.59 Real=18.19 Sys=1.79 e-/1-10: User=82.83 Real=98.62 Sys=7.45 Pedro Arce

In GAMOS we developed a tool to Physics cuts Optimization Geant4 has no default production thresholds (and the same for user limits: step size, kill below energy, …) It is important to optimise them You may be wasting your CPU time …or biasing your results But to optimize the cuts for different particle/regions need to run very big statistics need to run many sets of cut values log10(range) of gammas created at target that reach the patient or produce a secondary that reaches the patient CUT Percent Depth Dose curve for all particles and particles rejected by cut 1mm/e- 10 mm/gamma In GAMOS we developed a tool to Automatic obtain the best production cuts or user limits in one job with limited statistics Pedro Arce

General importance sampling Importance Sampling can be switched on Any particle Any data (also N-dimensional data space) Defining your own distribution In the distribution you assign a splitting value (SV) to each key value If SV > 1: particle will be split If SV < 1: Russian roulette will be played with the particle A parameter to avoid that particles with very low weights are split again and again Filter can be added, so that particles have to pass them before being split A new concept developed in GAMOS Pedro Arce

Other Optimization techniques Bias generation distribution Variables to bias: PosX, PosY, PosZ, PosPerp, PosR, PosPhi, PosTheta, DirTheta, DirPhi, Energy, Time User-defined distribution: P(V), V=Variable, P=Probability Kill selected particles with a user command Apply any filter (gammas; enter a volume; distance to a point; …) Geant4 biasing Cross Section change Force Collision Bremsstrahlung Splitting Plus two more efficient techniques: Directional bremsstrahlung splitting Equal-weight bremsstrahlung splitting Point detector technique When fraction of particles that reach your detector is many orders of magnitude far from what you can simulate Pedro Arce

Testing the code robustness Installation Each GAMOS release is tested to be correctly installed in three different Linux distributions and one MacOS Regression testing 165 tests are run to check the stability of the results with respect to previous releases Automatic statistical test using a Python-based utility User defines p-value for warning and error Comparison of over 3,000 variables and the analysis of over 8,000 binned Kolmogorov-Smirnov tests of histograms To help in flattening out the statistical fluctuations, each test is run 10 times ten times, and the average is computed. Color code: good / warning / bad p-value and test name Use it for your own simulation statistical tests: Check vs. experimental data Check two jobs with different simulation options Pedro Arce

+ 2,500 registered users since August ’09 Not only in medical physics Summary Software reliability has been a crucial part of the GAMOS developing cycle, trying to cover the different aspects of it. Moreover, its more than 2,000 registered users in different physics fields have greatly helped to improve the reliability of the code along the years. The GAMOS framework has demonstrated to be easy and flexible tool for Geant4 simulations New functionalities extend its use in several fields: DICOM management, ion therapy, tissue optics New protontherapy tutorial New tool to use GAMOS from a web browser Geographical distribution of GAMOS users + 2,500 registered users since August ’09 Not only in medical physics Physics fields of interest of GAMOS users An active community! +600 conversations with +2,000 messages in GAMOS User’s Discussion Forum Pedro Arce

3rd GAMOS online course: http://fismed.ciemat.es/GAMOS or Geant4 web (http://geant4.org)  Applications  Medical 3rd GAMOS online course: February 5th – March 2nd 2018