Geant beta Steps towards release 10 Gabriele Cosmo, PH/SFT

Multi-threading from prototype to production … Capitalizing the work started back in 2009 By X.Dong and G.Cooperman, Northeastern University Strong contribution by SFT Simulation team members Big effort brought to success 10.0-beta announced on June 28 th on schedule SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo2 G4MT 9.4 (2011) G4MT 9.5 (2012) G beta (now) G (Dec. 2013) G4 10 series (2014+) Proof of principle Identify objects to be shared First testing MT code integrated into G4 API re-design Examples migration Further testing First optimisations Public release All functionalities ported to MT Further refinements Focus on further performance improvements

Multi-threading 10.0-beta features - 1/2 Event-level parallelism Each worker thread proceeds independently Initializes its state from a master thread Identifies its part of the work (events) Generates hits in its own hits- collection Uses thread-private objects and state Shares read-only data structures (e.g. geometry, cross-sections, …) Has its own read-write part in a few ‘shared/split’ objects SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo3 Possibility to install/run Geant4 either in pure sequential or parallel (MT) mode Choice at configuration/installation time Sequential mode currently the default

Multi-threading 10.0-beta features - 2/2 SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo4 Focus on “lock-free” code Metrics currently in use: linearity of speed-up (w.r.t. #threads) Absolute throughput optimisation will follow Enforce use of POSIX standards to allow for integration with user preferred parallelization frameworks (e.g. TBB, MPI, …) See: Design aimed to minimize changes in users code Keep API changes at minimum

Multi-threading 10.0-beta known limitations Radioactive-decay and/or ion beams not yet ported Requires specific treatment for ion / isomers / decay of nuclei Some cases of event non-reproducibility to be investigated Goal: guarantee reproducibility at -numerical- level vs. sequential runs Visualization not yet fully functional Event/hits display during the event loop is possible only on some circumstances Some objects not cleanly deleted at the termination of the thread/program No proper sanity checking for memory leaks applied for 10.0-beta Some UI commands combinations have not been fully ported yet Limited testing coverage For physics/geometry options/phase-spaces SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo5

Multi-threading 10.0-beta performance - 1/3 Showing good efficiency w.r.t. perfect linearity (90%, 80% in HT) SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo6 (*) Based on performance analysis by S.Yung Jun, FNAL on AMD Opteron™ 6128, 32 cores No measured CPU degradation vs. sequential runs (*) Preliminary: CMS geometry HT regime Intel® Xeon® CPU 2.27GHz e-

Multi-threading 10.0-beta performance - 2/3 Hybrid mode: Host + Intel® Xeon Phi™ coprocessor (MIC) First look at total throughput (evt/s) (*) Excellent results: factor ~x3 in events produced w.r.t. host only SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo7 (*) Preliminary analysis on full-CMS benchmark by A.Dotti, SLAC Confirmed good scalability up to O(100) threads Reduced use of memory (see next slide)

Multi-threading 10.0-beta performance - 3/3 Hybrid mode: Host + Intel® Xeon Phi™ coprocessor Using out-of-the-box beta (i.e. no optimisations) 40 MB/thread Baseline: Full-CMS benchmark; 200 MB (geometry and physics) Speedup almost linear with reasonably small increase of memory usage SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo8 (*) Preliminary analysis on full-CMS benchmark by A.Dotti, SLAC Number of threads Memory usage (MB)

Multi-threading First physics validation results… 20 Gev proton on W-Lar FTFP_BERT physics-list Sequential: 5000 events Multi-threaded: events 4 threads results for 1 thread shown SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo9

Multi-threading Next to come … Review and further refinements to API Based on feedback from users and Beta testers Rationalisation and better modularisation of code for the initialisation of threads Aiming to further simplify user-code migration Address and solve current limitations & problems Improve testing coverage Further improve performance Identify and solve hotspots Use of thread-private malloc (to remove hidden locks in new/delete) Further investigations on task-based parallelism (TBB) TBB works already with Geant4-MT Provide one or more examples based on the new API Study heterogeneous parallelism (MPI together with multi-threading) Use in hybrid systems (host + one [or more] MIC card) Adoption of check-pointing technique (DMTCP) to improve start-up time SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo10

Electromagnetic Physics

Electromagnetic Physics 10.0-beta features - 1/2 Reviewed e+e- pair-production model Corrected inaccuracy of interpolation to improve e+e- spectra Implemented larger table Consolidation of multiple-scattering models Fixed long-standing issue with Urban93 Fine tuning of Urban96, now the most accurate for e- transport Updated WentzelVI and Single-scattering models, to use different screening parameters for e+e- and heavy particles; now the most accurate model for muons and hadrons Developed new validation tests Fluctuation models; tracker devices; electron ionisation SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo12

Electromagnetic Physics 10.0-beta features - 2/2 SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo13 ATLAS-barrel type calorimeter

Electromagnetic Physics Next to come … Implement full sharing of EM tables among threads Complete analysis of possible alternative models for fluctuation of energy loss Refinements to PAI model to be ready for use in production Refinements to effective charge approach for ion ionisation Combination of models for multiple and single scattering of hadrons Taking into account interference between Coulomb and strong amplitudes Summer student project Improvements to muon-nuclear cross-sections SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo14

Hadronic Physics

Hadronic Physics 10.0-beta features - 1/3 Fritiof string model (FTF) Extension to nucleus-nucleus interactions Now FTF (from ~3 GeV to ~TeV) together with QMD, or INCL++ or BIC (below a few GeV) allows to simulate ion-ion collisions for the first time in Geant4 Extended validation of hadron-nucleus interactions Improved excitation energies of nuclear residuals Bertini-like intra-nuclear cascade model (BERT) Improved two-body angular distributions effect on lateral shapes of hadronic showers Added nuclear capture of muons (which generate cascade) Revised treatment of cascade kinematics SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo16

Hadronic Physics 10.0-beta features - 2/3 SAID calculations of cos( Ɵ ) distributions for p-p elastic scattering Histograms fitted for interpolation (binned distribution of cos( Ɵ ) vs. Ekin) Below: hadronic showers in simplified calorimeters Wider lateral shapes SFT Group Meeting - 8 July

Hadronic Physics 10.0-beta features - 3/3 Pre-Compound and nuclear de-excitation Introduced the possibility to produce isomers Neutron High Precision (HP) Allow reading compressed data files INCL++ Improvement in the nucleus-nucleus sector Removed deprecated CHIPS classes and modules Major restructure of Physics Lists module and sub-modules Replaced residual dependencies on LEP/HEP models (parameterised, Geisha-like) in QGS-based lists with FTFP-BERT FTF will be replaced by GGS when this is extended to lower energies Removed several deprecated lists and added new INCL++- based physics-lists SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo18

Hadronic Physics Next to come … SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo19 Refinements to diffraction and code improvements to FTF Validation of pre-compound for all de-excitation in Bertini and enhancements to model for in-medium N-N cross-sections, physical unit parameters and coalescence Neutron High-Precision (HP): model review and extension to generic particles; extended validation of libraries and further comparisons with MCNP Cross-sections: extended validation and new general comprehensive test-suite Revision of Pre-Compound and Radioactive-decay for isomers

Geometry & more…

Geometry 10.0-beta features Replaced UI commands for geometry overlaps check Now based on built-in overlaps checking for random points generated on solids’ surfaces Using precise safety computation by default in navigation Archived obsolete BREPs classes and module SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo21

Geometry Next to come … Integration of the AIDA Unified Solids library To be included as optional component, for replacing the original solids SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo22

More features in 10.0-beta… Automatically generating isotope vector with natural abundances for NIST materials Variables shadowing … Units & constants inclusion Enhanced CMake build system Redesigned examples (basic & extended) Several examples migrated to support multi-threading New data sets G4EMLOW-6.33G4NDL-4.3G4NEUTRONXS-1.3G4RadioactiveDecay-3.7 SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo23

Physics Validation Enhanced tool for presentation of results Allowing for easy extension with new validation tests SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo24 Planned to complement validation on the GRID with new tests Tracker test LAr electro-magnetic calorimeter test Thin-target performance test

Supported platforms Geant beta Linux SLC6, gcc-4.4.7, 4.3.x, 64 bits MacOSX 10.7, 10.8, gcc-4.2.1, 64 bits Windows 7, Visual C (Visual Studio 2010) Multi-threading not ported yet ! Also tested: Linux SLC5, gcc-4.7, gcc-4.8, icc-13 Linux Ubuntu 12, gcc-4.6 Windows 7, VC (no MT port yet) SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo25

Summary Release 10.0-beta introduces ‘optional’ event-level parallelism through use of independent working threads Excellent scalability vs. #threads up to O(100) threads with no performance penalty vs. sequential mode First physics validation tests are positive We’re on track! … but still quite some work ahead of us to the final release for further improvements in testing coverage, performance and API optimisation Lots of new features in all areas and more to come before the final release in December Notes: Work plan: Improved physics validation testing suite SFT Group Meeting - 8 July 2013Geant beta: Steps towards release 10 - G.Cosmo26 Special thanks to A.Dotti (SLAC), S.Y.Jun (FNAL), M.Gayer, V.Ivantchenko, G.Lestaris, and A.Ribon for providing most of the material presented in these slides!