1. PMaC Performance Modeling and Characterization A Static Binary Instrumentation Threading Model for Fast Memory Trace Collection Michael Laurenzano 1, Joshua Peraza 1, Laura Carrington 1, Ananta Tiwari 1, William A. Ward 2, Roy Campbell 2 1 Performance Modeling and Characterization (PMaC) Laboratory, San Diego Supercomputer Center 2 High Performance Computing Modernization Program (HPCMP), United States Department of Defense

2. PMaC Performance Modeling and Characterization Memory-driven HPC  Many HPC applications are memory bound –Understanding application requires understanding memory behavior  Measurement? (e.g. timers or hardware counters) –Measuring at fine grain with reasonable overheads & transparently is HARD –How to get sufficient detail (e.g., reuse distance?)  Binary instrumentation –Obtains low-level details of address stream –Details are attached to specific structures within the application

3. PMaC Performance Modeling and Characterization Convolution Methods map Application Signatures to Machine Profiles produce performance prediction HPC Target System Characteristics of HPC system – Machine Profile HPC Target System Machine Profile – characterizations of the rates at which a machine can carry out fundamental operations Measured or projected via simple benchmarks on 1-2 nodes of the system HPC Application Requirements of HPC Application – Application Signature PMaC Performance/Energy Models Performance of Application on Target system HPC Application Application signature – detailed summaries of the fundamental operations to be carried out by the application Collected via trace tools Performance Model – a calculable expression of the runtime, efficiency, memory use, etc. of an HPC program on some machine

4. PMaC Performance Modeling and Characterization Runtime Overhead is a Big Deal  Real HPC applications –Relatively long runtimes: minutes, hours, days? –Lots of CPUS: O(10 7 ) in largest supercomputers –High slowdowns create problems  Too long for queue  Unsympathetic administrators/managers  Inconvenience  Unnecessarily use resources  PEBIL = PMaC’s Efficient Binary Instrumentation for x86/Linux

5. PMaC Performance Modeling and Characterization What’s New in PEBIL?  It can instrument multithreaded code –Developers use OpenMP and pthreads! –x86_64 only –Provide access to thread-local instrumentation data at runtime  Supports turning instrumentation on/off –Very lightweight operation –Swap nops with inserted instrumentation code at runtime –Overhead close to zero when all instrumentation is removed

6. PMaC Performance Modeling and Characterization Binary Instrumentation in HPC  Tuning and Analysis Utilities (TAU) – Dyninst and PEBIL  HPCToolkit – Dyninst  Open SpeedShop – Dyninst  Intel Parallel Studio – Pin  Memcheck memory bug detector – Valgrind valgrind –-leak-check=yes...  Understanding performance and energy  Many research projects (not just HPC) –BI used in 3000+ papers in the last 15 years

7. PMaC Performance Modeling and Characterization Binary Instrumentation Basics Memory Address Tracing Original Instrumented 0000c000 : c000: 48 89 7d f8 mov %rdi,-0x8(%rbp) c004: 5e pop %rsi c005: 75 f8 jne 0xc004 c007: c9 leaveq c008: c3 retq 0000c000 : c000: 48 89 7d f8 mov %rdi,-0x8(%rbp) c004: 5e pop %rsi c005: 75 f8 jne 0xc004 c007: c9 leaveq c008: c3 retq 0000c000 : c000: // compute -0x8(%rbp) and copy it to a buffer c008: 48 89 7d f8 mov %rdi,-0x8(%rbp) c00c: // compute (%rsp) and copy it to a buffer c014: 5e pop %rsi c015: 75 f8 jne 0xc00c c017: c9 leaveq c018: c3 retq 0000c000 : c000: // compute -0x8(%rbp) and copy it to a buffer c008: 48 89 7d f8 mov %rdi,-0x8(%rbp) c00c: // compute (%rsp) and copy it to a buffer c014: 5e pop %rsi c015: 75 f8 jne 0xc00c c017: c9 leaveq c018: c3 retq

8. PMaC Performance Modeling and Characterization Enter Multithreaded Apps  All threads use a single buffer? –Don’t need to know which thread is executing  A buffer for each thread? –Faster. No concurrency operations needed –More interesting. Per-thread behavior != average thread behavior  PEBIL uses the latter –Fast method for computing location of thread-local data –Cache that location in a register if possible 0000c000 : c000: // compute -0x8(%rbp) and copy it to a buffer c008: 48 89 7d f8 mov %rdi,-0x8(%rbp) c00c: // compute (%rsp) and copy it to a buffer c014: 5e pop %rsi c015: 75 f8 jne 0xc00c c017: c9 leaveq c018: c3 retq 0000c000 : c000: // compute -0x8(%rbp) and copy it to a buffer c008: 48 89 7d f8 mov %rdi,-0x8(%rbp) c00c: // compute (%rsp) and copy it to a buffer c014: 5e pop %rsi c015: 75 f8 jne 0xc00c c017: c9 leaveq c018: c3 retq

9. PMaC Performance Modeling and Characterization Thread-local Instrumentation Data in PEBIL  Provide a large table to each process (2M) –Each entry is a small pool of memory (32 bytes)  Must be VERY fast –Get thread id (1 instruction) –Simple hash of thread id (2 instructions) –Index table with hashed id (1 instruction)  Assume no collisions (so far so good) Hash Function thread 1 id thread 2 id thread 3 id thread 4 id Thread-local memory pools thread 4’s memory pool

10. PMaC Performance Modeling and Characterization Caching Thread-local Data  Cache the address of thread-local data –Dead registers are known at instrumentation time –Is there 1 register in a function which is dead everywhere?  Compute thread-local data address only at function [re]entry  Should use smaller scopes! (loops, blocks) Significant reductions

11. PMaC Performance Modeling and Characterization Other x86/Linux Binary Instrumentation Tool NameStatic or Dynamic Thread-local Data AccessThreading Overhead Runtime overhead Pin 1 DynamicRegister stolen from program, program JIT-compiled around that lost register Very lowMedium Dyninst 2 EitherCompute thread ID (layered function call) at every point HighVaries PEBIL 3 StaticTable + fast hash function (4 instructions), cache result in dead registers Low 1 Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation. Luk, C., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Vijay Janapa Reddi, and Hazelwood, K. ACM SIGPLAN Conference on Programming Language Design and Implementation, 2005. 2 An API for Runtime Code Patching. Buck, B. and Hollingsworth, J. International Journal of High Performance Computing Applications, 2000. 3 PEBIL: Efficient Static Binary Instrumentation for Linux. Laurenzano, M., Tikir, M., Carrington, L. and Snavely, A. International Symposium on the Performance Analysis of Systems and Software, 2010.

12. PMaC Performance Modeling and Characterization Runtime Overhead Experiments  Basic block counting  Classic test in binary instrumentation literature  Increment a counter each time a basic block is executed  Per-block, per-process, per-thread counters  Memory address tracing –Fill a process/thread-local buffer with memory addresses, then discard those addresses –Interval-based sampling  Take the first 10% of each billion memory accesses  Toggle instrumentation on/off when moving between sampling/non-sampling

13. PMaC Performance Modeling and Characterization Methodology  2 quad-core Xeon X3450, 2.67GHz –32K L1 and 256K L2 cache per core, 8M L3 per processor  NAS Parallel Benchmarks –2 sets: OpenMP and MPI, gcc/GOMP and gcc/mpich –8 threads/processes: CG, DC (omp only), EP, FT, IS, LU, MG –4 threads/processes: BT, SP  Dyninst 7.0 (dynamic) –Timing started when instrumented app begins running  Pin 2.12  PEBIL 2.0

14. PMaC Performance Modeling and Characterization Basic Block Counting (MPI)  All results are average of 3 runs  Slowdown relative to un-instrumented run –1 == no slowdown

15. PMaC Performance Modeling and Characterization Basic Block Counting (OpenMP)  Y-axis = log-scale slowdown factor  Dyninst thread ID lookup at every basic block

16. PMaC Performance Modeling and Characterization Threading Support Overhead (BB Counting)

17. PMaC Performance Modeling and Characterization Memory Tracing (MPI)  Slowdown relative to un-instrumented application ToolBTCGEPFTISLUMGSPMEAN PEBIL14.934.182.173.532.856.084.774.135.33 Pin5.894.432.913.893.544.674.852.484.08 Dyninst22.4413.765.649.537.2512.9015.3110.1912.12

18. PMaC Performance Modeling and Characterization Memory Tracing (OpenMP)  Instrumentation code inserted at every memory instruction –Dyninst computes thread ID at every memop –Pin runtime-optimizes instrumented code  Lots of opportunity to optimize ToolBTCGDCEPFTISLUMGSPMEAN PEBIL16.646.052.002.556.215.9410.5510.189.327.71 Pin6.194.423.01 3.855.565.895.264.774.66 Dyninst???862.86530.55448.89921.89752.181759.241555.76???975.90** 30s  7h45m

19. PMaC Performance Modeling and Characterization Interval-based Sampling  Extract useful information from a subset of the memory address stream –Simple approach: the first 10% of every billion addresses  In practice we use a window 100x as small –Obvious: avoid processing addresses (e.g., just collect and throw away) –Not so obvious: avoid collecting addresses  Instrumentation tools can disable/re-enable instrumentation –PEBIL: binary on/off. Very lightweight, but limited –Pin and Dyninst: arbitrary removal/reinstrumentation. Heavyweight, but versatile –Sampling only requires on/off functionality

20. PMaC Performance Modeling and Characterization Sampled Memory Tracing (MPI)  PEBIL always improves, and significantly  Pin usually, but not always improves –Amount and complexity of code re-instrumented during each interval probably drives this  Dyninst never improves –P ToolBTCGEPFTISLUMGSPMEAN PEBIL Full14.934.182.173.532.856.084.774.135.33 PEBIL 10%4.362.081.481.771.702.972.201.732.28 Pin Full5.894.432.913.893.544.674.852.484.08 Pin 10%5.425.012.612.743.195.054.512.983.93 Pin Best5.424.432.612.743.194.674.512.483.75 Dyninst Full = Best 22.4413.765.649.537.2512.9015.3110.1912.12

21. PMaC Performance Modeling and Characterization Sampled Memory Tracing (OpenMP) ToolBTCGDCEPFTISLUMGSPMEAN PEBIL16.646.052.002.556.215.9410.5510.189.327.71 PEBIL 10%4.592.751.781.592.432.613.293.363.292.85 Pin Full6.194.423.01 3.855.565.895.264.774.66 Pin 10%6.044.483.592.803.053.8810.266.478.965.50 Pin Best6.044.423.012.803.053.885.895.264.774.35 Dyninst Full = Best* ???862.86530.55448.89921.89752.181759.241555.76???975.90**

22. PMaC Performance Modeling and Characterization Conclusions  New PEBIL features –instrument multithreaded binaries –Turn instrumentation on/off  Fast access to per-thread memory pool to support per-thread data collection –Reasonable overheads  Cache memory pool location –Currently done at function level –Future work: smaller scopes  PEBIL is useful for practical memory address stream collection –Message passing or threaded

23. PMaC Performance Modeling and Characterization https://github.com/mlaurenzano/PEBIL michaell@sdsc.edu Questions?