1. Vector Prototype Status Philippe Canal (For VP team)

2. Components Scheduler UGeom Physics Simulation 7/21/14Vector Prototype Status2

3. About Scheduler Data structures Baskets and basket management Basket managers (per LV) Track and basket lifecycle Transport (physics and geometry) and track phases Scheduler workflow 7/21/14Vector Prototype Status3

4. GeantTrack Track identifiers – event, slot (memory management), track ID, PDG, G5 code Particle identifiers – PDG, GeantV code, charge, mass, species Kinematics – position, direction, momentum, energy Status flags – status, N steps, N null steps, boundary flag, pending flag Geometry/physics context – process, proposed step, current step, edep, distance to boundary, safety, current path, next path 7/21/14Vector Prototype Status4

5. GeantTrack fEvent fEvslot fParticle fPDG … fXpos fYpos fZpos fXdir fYdir fZdir … Edep Pstep Snext Safety fEvent fEvslot fParticle fPDG … fXpos fYpos fZpos fXdir fYdir fZdir … Edep Pstep Snext Safety *fPath *fNextpath *fEventV *fEvslotV *fParticleV *fPDGV … *fXposV *fYposV *fZposV *fXdirV *fYdirV *fZdirV … *fEdepV *fPstepV *fSnextV *fSafetyV *fEventV *fEvslotV *fParticleV *fPDGV … *fXposV *fYposV *fZposV *fXdirV *fYdirV *fZdirV … *fEdepV *fPstepV *fSnextV *fSafetyV 004080C0 00 *fPathV *fNextpath V fEventV fEvslotV fParticle V V fPDGV … … fXPosV fYPosV fZPosV fSnextV fSafetyV … … fPathV fNextpathV fNtracks=10 padding=32 vector 1 vector 2 GeantTrackPool GeantTrack GeantTrack_v SOA of fNtracks fBuffer 192 bytes TO FIX 7/21/14Vector Prototype Status5

6. Track_v operations (overhead) Pre-requirement to use vectorized: contiguity at the beginning of the arrays fEventVfParticleV … During transport, tracks stop leaving holes in the container Method(fXposV,…, fNtracks) or Method(GeantTrack_v &) fEventVfParticleV … fEventVfParticleV fEventVfParticleV fEventVfParticleV Use Compact Move A A A B A 7/21/14Vector Prototype Status6

7. Track_v operations (overhead) Track selection according some criteria fEventVfParticleV … Tracks have to be copied to a receiver during rescheduling fEventVfParticleV … fEventVfParticleV fEventVfParticleV Reshuffle Copy fEventVfParticleV … AA A B C Concurrency support 7/21/14Vector Prototype Status7

8. GeantBasket Elementary work unit for GeantV – They currently only hold tracks that are physically inside a given logical volume – Input GeantTrack_v array, filled by the scheduler – Output GeantTrack_v array, filled during transport Baskets have thread local access during transport, but concurrent access during scheduling Input Scheduler Transport Physics Output 7/21/14Vector Prototype Status8

9. Automatic basket scheduling Concurrent track addition, garbage collection, collection of tracks from prioritized events Adjustable threshold – T vol = N tracks_in_flight /2N threads rounded to %4 (min 4, max 256) Volume BM fThreshold * current empty Basket pool Transport queue GeantScheduler bottleneck 7/21/14Vector Prototype Status9

10. Basket lifecycle empty full Basket pool TGeoVolume Basket manager current Generator Scheduler 1…N volumes Transport queue Propagator transported recycle AddTrack priority AddTrack Push on threshold Push on garbage collection 1…N workers 7/21/14Vector Prototype Status10

11. Track lifecycle PhysicsSelect fProcessV[i] fPstepV[i] PropagateTracks Input tracks Output tracks kCrossing kExiting kPhysics kKilled (geom) PostStep (continuous) PostStep (continuous) fXposV[i], … fXdir[i], …, fPV[i], fEV[i] PostStep (discrete) PostStep (discrete) kNew kKilled(phys) kKilled(phys) 7/21/14Vector Prototype Status11

12. PropagateTracks kVector – continue in vector mode kSingle – call PropagateTracksSingle at the given stage kPostpone – copy remaining tracks to output MarkRemoved + Compact – compact holes and copy these tracks to the output PostponedAction kVector kSingle kPostpone ComputeTransport Length ComputeTransport Length FindNextBoundary AndStep FindNextBoundary AndStep vectorloop Propagate Neutrals Propagate Neutrals kCrossing kExiting kPhysics MarkRemoved Compact(output) MarkRemoved Compact(output) Propagate Safe<Pstep Propagate Safe<Pstep kPhysics Propagate Close to bound. Propagate Close to bound. kCrossing kExiting Propagate with safety Propagate with safety fSnextV[i], fSafetyV[i] stage0 stage1 stage2 7/21/14Vector Prototype Status12

13. Propagation to boundaries Safety-based approach algorithm very slow What is the step in magnetic field which shifts the final particle position with no more than epsilon with respect to linear propagation? – If proposed step within isotropic safety: use safety – Otherwise take into account only safe_step value in competition with distance to boundary and proposed step C =1/R ε = 1 micron safe_step = 2√ε/C 7/21/14Vector Prototype Status13

14. Track stages Imported Pending (threshol d) Queued for pickup Being transpor ted Queued to be dispatch ed Scheduled Basket manage r Transport queue Generator Basket transport Scheduler queue Scheduler dispatch Priority dispatch 7/21/14Vector Prototype Status14

15. Scheduler Pulls transported baskets, dispatches tracks to basket managers per volume – Not anymore! Applying policies to: – Provide work balancing (concurrency) – Keep memory under control – Keep the vectors up (most of the time) 7/21/14Vector Prototype Status15

16. Scheduler workflow Recycle transported baskets Event done? Digitize event ImportTracks Digitize event ImportTracks Last event done? EXIT Priority is ON? Y Y Last PE done? PE = prioritized event PE range = event number range for priority events Stop priority mode Y Y Queue flushed? Flush priority baskets Y Q size<min Adjust basket size Y Priority = ON PE range = (last,last+4) Priority = ON PE range = (last,last+4) Collect prioritized tracks (once) Empty Q? Garbage collect Y Check track counters Digitize transported events and Inject new events into released slot Priority mode: the scheduler puts all tracks from priority events to special baskets, injected them at every loop regardless the content Garbage collect mode when the queue is empty: inject every basket regardless the content 7/21/14Vector Prototype Status16

17. Monitoring Main bottleneck: GeantObjectPool::Borrow/Return 7/21/14Vector Prototype Status17

18. Performance 1000 events with 100 tracks each, measured on a 24- core dual socket E5-2695 v2 @ 2.40GHz (IVB). 7/21/14Vector Prototype Status18

19. Physics Simulation Strategy – Implement tabulated physics Backport to Geant4 as a single process (incorporating all implemented physics) Compare back ported Physics to regular G4 – Both physics performance and run-time performance Then compare VP with tabulated physics against G4 with tabulated physics – Implement vectorized physics Same scheme for verification 7/21/14Vector Prototype Status19

20. Vector Prototype Status20

21. Vector Prototype Status21

22. Vector Prototype Status22

23. Vector Prototype Status23

24. Vector Prototype Status24

25. Vector Prototype Status25

26. Vector Prototype Status26

27. Vector Prototype Status27

28. Vector Prototype Status28

29. Vector Prototype Status29

30. Vector Prototype Status30

31. Tabulated Physics Everything (except decay) is implemented both behind Geant4 (as a TotalPhysicsProcess) and behind VP Simple final state correction is implemented – scaling of the 3-momentum; of course not correct but we cannot do anything else for now. exampleN03 – exampleN03 can now be executed both by using the tabulated physics (default physics list TABPHYS) and FTFP BERT, FTFP BERT HP, QBBC physics lists. Physics list can be selected by -p flag at execution. – both production cuts(fixed to 1.0 [keV]) and tracking cuts are set (in energy) when exampleN03 is executed by using one of the original Geant4 physics lists – tracking cuts can be set by the -l flag at execution (both in case of G4 and TABPHYS physics lists) 7/21/14Vector Prototype Status31

32. Geant4+FTFP BERT vs Geant4+TABPHYS First results by using e − as primaries with energies of 30, 300, 3000, 30000 [MeV]: – production and tracking cuts are the same and set in energy – we don’t have range tables in the tabulated physics – linLossLimit is set to 100% in Geant4 – fluctuations and decay are switched off in Geant4 Energy grid of tabulation: – E p = 30, 300, 30000 [MeV]: 1000 bins between 1.0[keV ] − 3.0[GeV ] (logscale) and 10 final states – E p = 30 [GeV]: 100 bins between 1.0[keV ] − 1.0[TeV ] (logscale) and 5 final states 7/21/14Vector Prototype Status32

33. Next steps Further data will be generated by using Geant4 – switching on fluctuations – setting back linLossLimit to 1-2 % and so on to see these effects... – Decay Can start debugging the prototype! First by comparing these simple statistics generated by Geant4+TABPHYS and GeantV+TABPHYS 7/21/14Vector Prototype Status33

34. 7/21/14Vector Prototype Status34

35. 7/21/14Vector Prototype Status35

36. 7/21/14Vector Prototype Status36

37. 7/21/14Vector Prototype Status37

38. Geometry Vectorized Propagator implemented Merg(ing) with Usolids – Repository merged – Backward compatible interface Shapes – Box – Paraboloid – Trapezoid – Parallelepiped – Tube Coming soon – Hyperboloid – Polyhedra – Orb – TRD 7/21/14Vector Prototype Status38

39. 7/21/14Vector Prototype Status39

40. Trapezoid 7/21/14Vector Prototype Status40

41. 7/21/14Vector Prototype Status41

42. 7/21/14Vector Prototype Status42

43. 7/21/14Vector Prototype Status43

44. 7/21/14Vector Prototype Status44

45. Connecting Geant-V and VecGeom Geant-V could already use VecGeom in serial mode Geant-V can now use VecGeom in vector mode – added missing pieces in Geant-V: some thread local data to provide workspace – completed and tested vector navigation functionality in VecGeom – connected the two 7/21/14Vector Prototype Status45

46. First glance at performance Did some initial “valgrind --tool=callgrind” benchmarks of Geant-V Used scheduler version with hard working scheduler thread ( on 4+1 threads ) – 200 events – Ex03 geometry – SSE instructions 7/21/14Vector Prototype Status46

47. Major cpu users 13 % log 13 % Geometry Navigation 5 % memcpy 7/21/14Vector Prototype Status47

48. Current influence of VecGeom on overall performance 7/21/14Vector Prototype Status48

49. Geometry Integration Next Steps Performance tuning of vector navigation – global to local transformation not yet optimal – a couple of other ideas comparison to tabulated Geant4 simulation gradually more complicated geometries ( we should now put tubes, traps,... ) 7/21/14Vector Prototype Status49

50. Summary Progress on all 3 parts in both: – Performance – Breadth Moving along toward silver bullet measurement 7/21/14Vector Prototype Status50