IceCube simulation with PPC Photonics: 2000 – up to now Photon propagation code PPC: now

Photonics: conventional on CPU First, run photonics to fill space with photons, tabulate the result Create such tables for nominal light sources: cascade and uniform half-muon Simulate photon propagation by looking up photon density in tabulated distributions  Table generation is slow  Simulation suffers from a wide range of binning artifacts  Simulation is also slow! (most time is spent loading the tables)

Direct photon tracking with PPC simulating flasher/standard candle photons same code for muon/cascade simulation using precise scattering function: linear combination of HG+SAM using tabulated (in 10 m depth slices) layered ice structure employing 6-parameter ice model to extrapolate in wavelength tilt in the ice layer structure is properly taken into account transparent folding of acceptance and efficiencies precise tracking through layers of ice, no interpolation needed precise simulation of the longitudinal development of cascades and angular distribution of particles emitting Cherenkov photons photon propagation code

PPC simulation on GPU graphics processing unit execution threads propagation steps (between scatterings) photon absorbed new photon created (taken from the pool) threads complete their execution (no more photons) Running on an NVidia GTX 295 CUDA-capable card, ppc is configured with: 448 threads in 30 blocks (total of threads) average of ~ 1024 photons per thread (total of photons per call)

Photon Propagation Code: PPC There are 5 versions of the ppc: original c++ "fast" c++ in Assembly for CUDA GPU icetray module All versions verified to produce identical results comparison with i3mcml

ppc icetray module at uses a wrapper: private/ppc/i3ppc.cxx, which compiles by cmake system into the libppc.so it is necessary to compile an additional library libxppc.so by running make in private/ppc/gpu:  “make glib” compiles gpu-accelerated version (needs cuda tools)  “make clib” compiles cpu version (from the same sources!) link to libxppc.so and libcudart.so (if gpu version) from build/lib directory this library file must be loaded before the libppc.so wrapper library  These steps are automated with a resouces/make.sh script

ppc homepage

GPU scaling Original:1/2.081/2.70 CPU c++: Assembly: GTX 295: GTX/Ori: C1060: C2050: GTX 480: On GTX 295: GHz Running on 30 MPs x 448 threads Kernel uses: l=0 r=35 s=8176 c=62400 On GTX 480: GHz Running on 15 MPs x 768 threads Kernel uses: l=0 r=40 s=3960 c=62400 On C1060: GHz Running on 30 MPs x 448 threads Kernel uses: l=0 r=35 s=3992 c=62400 On C2050: GHz Running on 14 MPs x 768 threads Kernel uses: l=0 r=41 s=3960 c=62400 Uses cudaGetDeviceProperties() to get the number of multiprocessors, Uses cudaFuncGetAttributes() to get the maximum number of threads

GTX 480 vs. GTX 295 GTX 295 has 2 GPUs 240 MPs in 30 cores 8 MPs per core 2 single-precision sFPUs  60 sFPUs per GPU  480 cores per card  120 sFPUs per card Shared memory: 16Kb per core 960 Kb per card GTX 480 has 1 GPU 480 MPs in 15 cores 32 MPs per core 4 single-precision sFPUs  60 sFPUs per GPU  480 cores per card  60 sFPUs per card (!) shared memory up to 48Kb of per core up to 720 Kb per card Why is ppc not a factor 2 faster on GTX 480 GPU than on GTX 295 GPU?

Device enumeration cudatest: Found 6 devices, driver 2030, runtime (1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l1 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 1(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 2(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 3(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 4(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 5(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) 3 GTX 295 cards, each with 2 GPUs PSU 0 and 1 4 and 5 2 and 3 nvidia-smi -lsa GPU 0: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 87 C GPU 1: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 90 C GPU 2: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 100 C GPU 3: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 105 C GPU 4: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 100 C GPU 5: Product Name: GeForce GTX 295 Serial: PCI ID: 5eb10de Temperature: 103 C

Device enumeration cuda002: Found 5 devices, driver 3010, runtime (2.0): GeForce GTX GHz G( ) S(49152) C(65536) R(32768) W(32) l0 o1 c0 h1 i0 m15 a512 M( ) T(1024: 1024,1024,64) G(65535,65535,1) 1(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M( ) T(512: 512,512,64) G(65535,65535,1) 2(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M( ) T(512: 512,512,64) G(65535,65535,1) 3(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M( ) T(512: 512,512,64) G(65535,65535,1) 4(1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l0 o1 c0 h1 i0 m30 a256 M( ) T(512: 512,512,64) G(65535,65535,1) 2 GTX 295 cards, 1 GTX 480 card PSU 1 and 2 3 and 4 0 nvidia-smi -a GPU 0: Product Name: GeForce GTX 295 PCI ID: 5eb10de Temperature: 68 C GPU 1: Product Name: GeForce GTX 295 PCI ID: 5eb10de Temperature: 73 C GPU 2: Product Name: GeForce GTX 480 PCI ID: 6c010de Temperature: 106 C GPU 3: Product Name: GeForce GTX 295 PCI ID: 5eb10de Temperature: 90 C GPU 4: Product Name: GeForce GTX 295 PCI ID: 5eb10de Temperature: 91 C 0 and 1 3 and 4 2

Fermi vs. Tesla cudatest: Found 6 devices, driver 2030, runtime (1.3): GeForce GTX GHz G( ) S(16384) C(65536) R(16384) W(32) l1 o1 c0 h1 i0 m30 a256 M(262144) T(512: 512,512,64) G(65535,65535,1) tesla: Found 1 devices, driver 3000, runtime (1.3): Tesla C GHz G( ) S(16384) C(65536) R(16384) W(32) l1 o1 c0 h1 i0 m30 a256 M( ) T(512: 512,512,64) G(65535,65535,1) fermi: Found 1 devices, driver 3000, runtime (2.0): Tesla C GHz G( ) S(49152) C(65536) R(32768) W(32) l1 o1 c0 h1 i0 m14 a512 M( ) T(1024: 1024,1024,64) G(65535,65535,1) beta: Found 1 devices, driver 3010, runtime (2.0): Tesla C GHz G( ) S(49152) C(65536) R(32768) W(32) l1 o1 c0 h1 i0 m14 a512 M( ) T(1024: 1024,1024,64) G(65535,65535,1) 11:arch=11 make gpu 12:arch=12 make gpu (default/best) 1x:arch=12 make gpu with -ftz=true -prec-div=false -prec-sqrt=false 20:arch=20 make gpu 2x:arch=20 make gpu with -ftz=true -prec-div=false -prec-sqrt=false Flasherf2kmuon

Kernel time calculation Run 3232 (corsika) IC86 processing on cuda002 (per file): GTX 295: Device time: (in-kernel: ) [ms] GTX 480: Device time: (in-kernel: ) [ms] If more than 1 thread is running using same GPU: Device time: (in-kernel: ) [ms] 3 counters:1. time difference before/after kernel launch in host code 2. in-kernel, using cycle counter:min thread time 3.max thread time Also, real/user/sys times of top: gpus6 cpus1 cores8 files693 Real749m4.693s User3456m10.888s sys39m50.369s Device time: [ms] files: 693 real: user: gpu: kernel: [seconds] 81%-91% GPU utilization

Concurrent execution time CPUGPUCPUGPU Thread 1: CPUGPUCPUGPU Thread 2: CPU GPU CPU GPU CPU GPU CPU GPU One thread: Create track segments Copy track segments to GPU Process photon hits Copy photon hits from GPU Need 2 buffers for track segments and photon hits However: have 2 buffers: 1 on host and 1 on GPU! Just need to synchronize before the buffers are re-used

BAD multiprocessors (MPs) clist cudatest cuda cuda cuda #badmps cuda cuda cuda Disable 3 bad GPUs out of 24: 12.5% Disable 3 bad MPs out of 720: 0.4%! Configured: xR=5 eff=0.95 sf=0.2 g=0.943 Loaded 12 angsens coefficients Loaded 6x170 dust layer points Loaded random multipliers Loaded 42 wavelenth points Loaded 171 ice layers Loaded 3540 DOMs (19x19) Processing f2k muons from stdin on device 2 Total GPU memory usage: photons: hits: 991 Error: TOT was a nan or an inf 1 times! Bad MP #20 photons: hits: 393 photons: hits: 570 photons: hits: 501 photons: hits: 832 photons: hits: 717 CUDA Error: unspecified launch failure Total GPU memory usage: photons: hits: 938 Error: TOT was a nan or an inf 9 times! Bad MP #20 #20 #20 #20 photons: hits: 442 photons: hits: 627 CUDA Error: unspecified launch failure gpu]$ cat mmc.1.f2k | BADMP=20./ppc 2 > /dev/null Configured: xR=5 eff=0.95 sf=0.2 g=0.943 Loaded 12 angsens coefficients Loaded 6x170 dust layer points Loaded random multipliers Loaded 42 wavelenth points Loaded 171 ice layers Loaded 3540 DOMs (19x19) Processing f2k muons from stdin on device 2 Not using MP #20 Total GPU memory usage: photons: hits: 871 … photons: hits: 114 Device time: (in-kernel: ) [ms] Failure rates:

Typical run times corsika: run 3232: sec files ic86/spx/3232 on cuda00[123] (53.4 seconds per job) 1.2 days of real detector time in 6.5 days nugen: run 2972: event files; E^-2 weighted ic86/spx/2972 on cudatest (25.1 seconds per job) entire 10k set of files in 2.9 days  this is enough for an atmnu/diffuse analysis! Considerations: Maximize GPU utilization by running only mmc+ppc parts on the GPU nodes still, IC40 mmc+ppc+detector was run with ~80% GPU utilization run with 100% DOM efficiency, save all ppc events with at least 1 MC hit apply a range of allowed efficiencies (70-100%) later with ppc-eff module

Other Consider: building production computers with only 2 cards, leaving a space in between using 6-core CPUs if paired with 3 GPU cards 4-way Tesla GPU-only servers a possible solution Consumer GTX card much faster than Tesla/Fermi cards GTX 295 was so far found to be a better choice than GTX 480 but: no loger available! Reliability: 0.4% loss of advertised capacity in GTX 295 cards however: 2 of 3 affected cards were “refurbished” do cards deteriorate over time? The failed MPs did not change in ~3 months