Shirley Moore shirley@cs.utk.edu Towards Scalable Cross-Platform Application Performance Analysis -- Tool Goals and Progress Shirley Moore shirley@cs.utk.edu October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Scalability Issues Code instrumentation Hand instrumentation too tedious for large codes Runtime control of data collection Batch queueing systems Cause problems for interactive tools Tracefile size and complexity Data analysis October 18, 2001 LACSI Symposium, Santa Fe, NM

Cross-platform Issues Goal: similar user interfaces across different platforms Tools necessarily rely on platform-dependent substrates – e.g., for accessing hardware counters. Standardization of interfaces and data formats promotes interoperability and allows design of portable tools. October 18, 2001 LACSI Symposium, Santa Fe, NM

Where is Standardization Needed? Performance data Trace records vs. summary statistics Data format Data semantics Library interfaces Access to hardware counters Statistical profiling Dynamic instrumentation October 18, 2001 LACSI Symposium, Santa Fe, NM

Standardization? (cont.) User interfaces Common set of commands Common functionality Timing routines Memory utilization information October 18, 2001 LACSI Symposium, Santa Fe, NM

Parallel Tools Consortium http://www.ptools.org/ Interaction between vendors, researchers, and users Venue for standardization Current projects PAPI DPCL October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Hardware Counters Small set of registers that count events, which are occurrences of specific signals related to the processor’s function Monitoring these events facilitates correlation between the structure of the source/object code and the efficiency of the mapping of that code to the underlying architecture. October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Goals of PAPI Solid foundation for cross platform performance analysis tools Free tool developers from re-implementing counter access Standardization between vendors, academics and users Encourage vendors to provide hardware and OS support for counter access Reference implementations for a number of HPC architectures Well documented and easy to use October 18, 2001 LACSI Symposium, Santa Fe, NM

PAPI Implementation Tools!!! PAPI High Level PAPI Low Level Portable Hardware Performance Counter Operating System Kernel Extension PAPI Machine Dependent Substrate Machine Specific Layer Portable October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM PAPI Preset Events Proposed standard set of events deemed most relevant for application performance tuning Defined in papiStdEventDefs.h Mapped to native events on a given platform Run tests/avail to see list of PAPI preset events available on a platform October 18, 2001 LACSI Symposium, Santa Fe, NM

Statistical Profiling PAPI provides support for execution profiling based on any counter event. PAPI_profil() creates a histogram by text address of overflow counts for a specified region of the application code. Used in vprof tool from Sandia Lab October 18, 2001 LACSI Symposium, Santa Fe, NM

PAPI Reference Implementations Linux/x86, Windows 2000 Requires patch to Linux kernel, driver for Windows Linux/IA-64 Sun Solaris 2.8/Ultra I/II IBM AIX 4.3+/Power Contact IBM for pmtoolkit SGI IRIX/MIPS Compaq Tru64/Alpha Ev6 & Ev67 Requires OS device driver patch from Compaq Per-thread and per-process counts not possible Extremely limited number of events Cray T3E/Unicos October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM PAPI Future Work Improve accuracy of hardware counter and statistical profiling data Microbenchmarks to measure accuracy (Pat Teller, UTEP) Use hardware support for overflow interrupts Use Event Address Registers (EARs) where available Data structure based performance counters (collaboration with UMd) Qualify event counting by address range Page level counters in cache coherence hardware October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM PAPI Future (cont.) Memory utilization extensions (following list suggested by Jack Horner, LANL) Memory available on a node Total memory available/used High-water-mark memory used by process/thread Disk swapping by process Process-memory locality Location of memory used by an object Dynamic instrumentation – e.g., PAPI probe modules October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM For More Information http://icl.cs.utk.edu/projects/papi/ Software and documentation Reference materials Papers and presentations Third-party tools Mailing lists October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM DPCL Dynamic Probe Class Library Built of top of IBM version of University of Maryland’s dyninst Current platforms IBM AIX Linux/x86 (limited functionality) Dyninst ported to more platforms but by itself lacks functionality for easily instrumenting parallel applications. October 18, 2001 LACSI Symposium, Santa Fe, NM

Infrastructure Components? Parsers for common languages Access to hardware counter data Communication behavior instrumentation and analysis Dynamic instrumentation capability Runtime control of data collection and analysis Performance data management October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Case Studies Test tools on large-scale applications in production environment Reveal limitations of tools and point out areas where improvements are needed Develop performance tuning methodologies for large-scale codes October 18, 2001 LACSI Symposium, Santa Fe, NM

PERC: Performance Evaluation Research Center Developing a science for understanding performance of scientific applications on high-end computer systems. Developing engineering strategies for improving performance on these systems. DOE Labs: ANL, LBNL, LLNL, ORNL Universities: UCSD, UI-UC, UMD, UTK Funded by SciDAC: Scientific Discovery through Advanced Computing October 18, 2001 LACSI Symposium, Santa Fe, NM

PERC: Real-World Applications High Energy and Nuclear Physics Shedding New Light on Exploding Stars: Terascale Simulations of Neutrino-Driven SuperNovae and Their NucleoSynthesis Advanced Computing for 21st Century Accelerator Science and Technology Biology and Environmental Research Collaborative Design and Development of the Community Climate System Model for Terascale Computers Fusion Energy Sciences Numerical Computation of Wave-Plasma Interactions in Multi-dimensional Systems Advanced Scientific Computing Terascale Optimal PDE Solvers (TOPS) Applied Partial Differential Equations Center (APDEC) Scientific Data Management (SDM) Chemical Sciences Accurate Properties for Open-Shell States of Large Molecules …and more… October 18, 2001 LACSI Symposium, Santa Fe, NM

Parallel Climate Transition Model Components for Ocean, Atmosphere, Sea Ice, Land Surface and River Transport Developed by Warren Washington’s group at NCAR POP: Parallel Ocean Program from LANL CCM3: Community Climate Model 3.2 from NCAR including LSM: Land Surface Model ICE: CICE from LANL and CCSM from NCAR RTM: River Transport Module from UT Austin Fortran 90 with MPI October 18, 2001 LACSI Symposium, Santa Fe, NM

PCTM: Parallel Climate Transition Model River Model Ocean Model Atmosphere Model Flux Coupler Land Surface Model Sea Ice Model Sequential Execution of Parallelized Modules October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM PCTM Instrumentation Vampir tracefile in tens of gigabytes range even for toy problem Hand instrumentation with PAPI tedious UIUC working on SvPablo instrumentation Must work in batch queueing environment Plan to try other tools MPE logging and jumpshot TAU VGV? October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM In Progress Standardization and reference implementations for memory utilization information (funded by DoD HPCMP PET, Ptools-sponsored project) Repositories of application performance evaluation case studies (e.g., SciDAC PERC) Portable dynamic instrumentation for parallel applications (DOE MICS project – UTK, UMd, UWisc) Increased functionality and accuracy of hardware counter data collection (DoD HPCMP, DOE MICS) October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Next Steps Additional areas for standardization? Scalable trace file format Metadata standards for performance data New hardware counter metrics (e.g., SMP and DMP events, data-centric counters) Others? October 18, 2001 LACSI Symposium, Santa Fe, NM

LACSI Symposium, Santa Fe, NM Next Steps (cont.) Sharing of tools and data Open source software Machine and software profiles Runtime performance data Benchmark results Application examples and case studies Long-term goal: common performance tool infrastructure across HPC systems October 18, 2001 LACSI Symposium, Santa Fe, NM