GeantV prototype at a glance A.Gheata Simulation weekly meeting July 8, 2014

Content Data structures Baskets and basket management Basket managers (per LV) Track and basket lifecycle Transport (physics and geometry) and track phases Scheduler workflow

GeantTrack Track identifiers – event, slot (memory management), track ID, PDG, G5 code Particle identifiers – PDG, GeantV code, charge, mass, species Kinematics – position, direction, momentum, energy Status flags – status, N steps, N null steps, boundary flag, pending flag Geometry/physics context – process, proposed step, current step, edep, distance to boundary, safety, current path, next path

GeantTrack fEvent fEvslot fParticle fPDG … fXpos fYpos fZpos fXdir fYdir fZdir … Edep Pstep Snext Safety fEvent fEvslot fParticle fPDG … fXpos fYpos fZpos fXdir fYdir fZdir … Edep Pstep Snext Safety *fPath *fNextpath *fEventV *fEvslotV *fParticleV *fPDGV … *fXposV *fYposV *fZposV *fXdirV *fYdirV *fZdirV … *fEdepV *fPstepV *fSnextV *fSafetyV *fEventV *fEvslotV *fParticleV *fPDGV … *fXposV *fYposV *fZposV *fXdirV *fYdirV *fZdirV … *fEdepV *fPstepV *fSnextV *fSafetyV C0 00 *fPathV *fNextpath V fEventV fEvslotV fParticle V V fPDGV … … fXPosV fYPosV fZPosV fSnextV fSafetyV … … fPathV fNextpathV fNtracks=10 padding=32 vector 1 vector 2 GeantTrackPool GeantTrack GeantTrack_v SOA of fNtracks fBuffer 192 bytes TO FIX

Track_v operations (overhead) Pre-requirement to use vectorized: contiguity at the beginning of the arrays fEventVfParticleV … During transport, tracks stop leaving holes in the container Method(fXposV,…, fNtracks) or Method(GeantTrack_v &) fEventVfParticleV … fEventVfParticleV fEventVfParticleV fEventVfParticleV Use Compact Move A A A B A

Track_v operations (overhead) Track selection according some criteria fEventVfParticleV … Tracks have to be copied to a receiver during rescheduling fEventVfParticleV … fEventVfParticleV fEventVfParticleV Reshuffle Copy fEventVfParticleV … AA A B C Concurrency support

GeantBasket Elementary work unit for GeantV – They currently only hold tracks that are physically inside a given logical volume – Input GeantTrack_v array, filled by the scheduler – Output GeantTrack_v array, filled during transport Baskets have thread local access during transport, but concurrent access during scheduling Input Scheduler Transport Physics Output

Automatic basket scheduling Concurrent track addition, garbage collection, collection of tracks from prioritized events Adjustable threshold – T vol = N tracks_in_flight /2N threads rounded to %4 (min 4, max 256) Volume BM fThreshold * current empty Basket pool Transport queue GeantScheduler bottleneck

Basket lifecycle empty full Basket pool TGeoVolume Basket manager current Generator Scheduler 1…N volumes Transport queue Propagator transported recycle AddTrack priority AddTrack Push on threshold Push on garbage collection 1…N workers

Track lifecycle PhysicsSelect fProcessV[i] fPstepV[i] PropagateTracks Input tracks Output tracks kCrossing kExiting kPhysics kKilled (geom) PostStep (continuous) PostStep (continuous) fXposV[i], … fXdir[i], …, fPV[i], fEV[i] PostStep (discrete) PostStep (discrete) kNew kKilled(phys) kKilled(phys)

PropagateTracks kVector – continue in vector mode kSingle – call PropagateTracksSingle at the given stage kPostpone – copy remaining tracks to output MarkRemoved + Compact – compact holes and copy these tracks to the output PostponedAction kVector kSingle kPostpone ComputeTransport Length ComputeTransport Length FindNextBoundary AndStep FindNextBoundary AndStep vectorloop Propagate Neutrals Propagate Neutrals kCrossing kExiting kPhysics MarkRemoved Compact(output) MarkRemoved Compact(output) Propagate Safe<Pstep Propagate Safe<Pstep kPhysics Propagate Close to bound. Propagate Close to bound. kCrossing kExiting Propagate with safety Propagate with safety fSnextV[i], fSafetyV[i] stage0 stage1 stage2

Propagation to boundaries Safety-based approach algorithm very slow What is the step in magnetic field which shifts the final particle position with no more than epsilon with respect to linear propagation? – If proposed step within isotropic safety: use safety – Otherwise take into account only safe_step value in competition with distance to boundary and proposed step C =1/R ε = 1 micron safe_step = 2√ε/C

Track stages Imported Pending (threshol d) Queued for pickup Being transpor ted Queued to be dispatch ed Scheduled Basket manage r Transport queue Generator Basket transport Scheduler queue Scheduler dispatch Priority dispatch

Scheduler Pulls transported baskets, dispatches tracks to basket managers per volume – Not anymore! Applying policies to: – Provide work balancing (concurrency) – Keep memory under control – Keep the vectors up (most of the time)

Scheduler workflow Recycle transported baskets Event done? Digitize event ImportTracks Digitize event ImportTracks Last event done? EXIT Priority is ON? Y Y Last PE done? PE = prioritized event PE range = event number range for priority events Stop priority mode Y Y Queue flushed? Flush priority baskets Y Q size<min Adjust basket size Y Priority = ON PE range = (last,last+4) Priority = ON PE range = (last,last+4) Collect prioritized tracks (once) Empty Q? Garbage collect Y Check track counters Digitize transported events and Inject new events into released slot Priority mode: the scheduler puts all tracks from priority events to special baskets, injected them at every loop regardless the content Garbage collect mode when the queue is empty: inject every basket regardless the content

Monitoring Main bottleneck: GeantObjectPool::Borrow/Return