Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Key Computation Issues Large volume of data ( disk / memory / network ) Significant number of solvers iterations due to numerical intractability Redundant memory accesses coming from interleaving data dependencies Use of double precision because of accuracy need (hardware penalty) Misaligned data (inherent to specific data structures) Exacerbates cache misses (depending on cache size) Becomes a serious problem when consider accelarators Leads to « false sharing » with Shared-Memory paradigm (Posix, OpenMP) Padding is one solution but would dramatically increase memory requirement Memory/Computation compromise in data organization (e.g. gauge replication)

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Why the CELL Processor ? Highest computing power in a single « computing node » Fast memory access Asynchronysm between data transfers and computation Issues with the CELL Processor ? Data alignment (both for calculations and transfers) Heavy use of list DMA Small size of the Local Store (SPU local memory) Ressources sharing with Dual Cell Based Blade Integration into an existing standard framework

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 What we have done Implementation of each critical kernel on the CELL processor SIMD version of basic operators Appropriate DMA mechanism (efficient list DMA and double buffering) Merging of consecutive operations into a unique operator ( latency & memory reuse ) Aggregation of all these implementations into a single and standalone library Effective integration into the tmLQCD package Successful tests (QS20 and QS22) A single SPU thread holds the whole set of routines SPU thread remains « permanently » active during a working session Data re-alignment Routine calls replacement (invoke CELL versions in place of native ones) This should be the way to commit this back to tmLQCD (external library and « IsCELL » switch)

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Global Organization Task partitioning, distribution, and synchorization are done by the PPU Each SPE operates on its data portion by a typical loop of the form (DMA get + SIMD Computation + DMA put) The SPE, always active, switches to the appropriate operation on each request

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Optimal list DMA organization for the Wilson-Dirac Operator The computation of Wilson-Dirac action for a set of K contigous spinors required to get 8K spinors ( Example below with 32x16 3 lattice and even-odd ) S[0]P[2048]P[63488]P[128]P[1920]P[8]P[120]P[0]P[7] S[1]P[2049]P[63489]P[129]P[1921]P[9]P[121]P[1]P[0] S[2]P[2050]P[63490]P[130]P[1922]P[10]P[122]P[2]P[1] S[3]P[2051]P[63431]P[131]P[1923]P[11]P[123]P[3]P[2] A direct list DMA to get this « spinors matrix » involves 8x4 DMA items A list DMA to get the « transpose » involves = 9 DMA items Generally, our list DMA is of size 8 + c k instead of 8K ( bin packing ) No impact on SPU performance because of the uniform access to the LS Significant improvment in global performance and scalability

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Performance results We consider a 32x16 3 lattice and CELL-accerated version of tmLQCD QS20 #SPETime(s)SpeedupGFlops QS22 #SPETime(s)SpeedupGFlops INTEL i7 quadcore 2.83 Ghz Without SSEWith SSE 1 core4 cores1 core4 cores INTEL i7CELL (8 SPEs) SSE + 4cQS20QS22 GCR (57 iters)11.05 s3.78 s2.04 s CG (685 iters)89.54 s42.25 s22.78 s

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Comments We observed a factor 2 between QS20 and QS22 We observed a factor 4 between QS22 and Intel i7 quadcore 2.83 Ghz Good scalability on QS20 Scalability on QS22 is alterated beyond 4 SPEs (probably a binding issue on the Dual Cell Based Blade, which should be easy to fix) Fixing this scalability issue on QS22 will double actual performances

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 Ways for improvment Implement the « non GAUGE COPY » version ( significant memory reduction / packing ) Outstanding performance expected Explore the SU(3) reconstruct approach at SPE level ( memory and bandwith savings ) Having the PPU participate in the calculations ( makes sens in double precision ) Try to scale up to the 16 SPEs on the QS22 Dual Cell Based Blade Experiment with a cluster of CELL processors

Aurora/PetaQCD/QPACE Metting Regensburg University, April 14-15, 2010 END