Update on G5 prototype Andrei Gheata Computing Upgrade Weekly Meeting 26 June 2012

Simple observation: HEP transport is mostly local ! ATLAS volumes sorted by transport time. The same behavior is observed for most HEP geometries. 50 per cent of the time spent in 50/7100 volumes

A playground for new ideas Simple simulation prototype to help exploring parallelism and efficiency issues – Basic idea: minimal physics to start with, realistic geometry: can we implement a parallel transport model on threads exploiting data locality and vectorisation? – Clean re-design of data structures and steering to easily exploit parallel architectures – Can we make it fully non-blocking from generation to digitization and I/O ? Events and primary tracks are independent – Work chunk: basket containing a vector of tracks – Mixing tracks from different events to avoid tails and have reasonably-sized vectors – Study how does scattering/gathering of vectors impact the simulation data flow Toy physics at first, more realistic EM & hadronic processes to continue with – The application should be eventually tuned based on realistic numbers New transport model more “detector element”-oriented, profiting from the cached data structures – geometry and x-section wise Where to go from there – Re-design the particle stack and the I/O – Re-design transport models from a “plug-in” perspective E.g. ability to use fast simulation on per track basis – Understand what can be gained and how, what is the impact on the existing code, what are the changes and effort to migrate to a new system…

Volume-oriented transport model We implemented a model where all particles traversing a given geometry volume are transported together as a vector until the volume gets empty – Same volume -> local (vs. global) geometry navigation, same material and same cross sections – Load balancing: distribute all particles from a volume type into smaller work units called baskets, give a basket to a transport thread at a time Particles exiting a volume are distributed to baskets of the neighbor volumes until exiting the setup or disappearing – Like a champagne cascade, but lower glasses can also fill top ones… – No direct communication between threads to avoid synchronization issues

The beginning Realistic geometry + event generator Inject event in the volume containing the IP More events better to cut event tails and fill better the pipeline !

A first approach Work queue Scatter all injected tracks to baskets. Only baskets above some threshold are transported. Transport threads pick-up baskets from the work queue Physics processes Geometry transport Particles(i 0,…,i n ) Physics processes and geometry transport called with vectors of particles Each thread transports its basket of tracks to the boundaries of the current volume Move crossing tracks to a buffer, then picks-up the next basket from the queue

First version required synchronization… Work queue POP_CHUNK QUEUE_EMPTY ParticleBuffer FLUSH Generation = Pop work chunks until the queue is empty Synchronization point: flush transported particle buffer and sort baskets according content Recompute work chunks and start transporting the next generation of baskets

Processing phases Initial events injection Optimal regime Constant basket content Sparse regime More and more frequent garbage collections Less tracks per basket Garbage collection threshold Depletion regime Continuous garbage collection New events needed Force flushing some events ideal

Prototype implementation transport pick-up baskets transportable baskets recycled baskets full track collections recycled track collections Worker threads Dispatch & garbage collect thread Crossing tracks (itrack, ivolume) Push/replace collection Main scheduler n Inject priority baskets recycle basket ivolume loop tracks and push to baskets n Stepping(tid, &tracks) Digitize & I/O thread Priority baskets Generate(N events ) Hits Hits Digitize(iev) Disk Inject/replace baskets deque generate flush

Evolution of populations Flush events

Preliminary benchmarks HT mode Excellent CPU usage Benchmarking 10+1 threads on a 12 core Xeon Locks and waits: some overhead due to transitions coming from exchanging baskets via concurrent queues Event re-injection will improve the speed-up