1. Shirley Moore shirley@cs.utk.edu
Towards Scalable Cross-Platform Application Performance Analysis -- Tool Goals and Progress
Shirley Moore
October 18, 2001
LACSI Symposium, Santa Fe, NM

2. LACSI Symposium, Santa Fe, NM
Scalability Issues
Code instrumentation
Hand instrumentation too tedious for large codes
Runtime control of data collection
Batch queueing systems
Cause problems for interactive tools
Tracefile size and complexity
Data analysis
October 18, 2001
LACSI Symposium, Santa Fe, NM

3. Cross-platform Issues
Goal: similar user interfaces across different platforms
Tools necessarily rely on platform-dependent substrates – e.g., for accessing hardware counters.
Standardization of interfaces and data formats promotes interoperability and allows design of portable tools.
October 18, 2001
LACSI Symposium, Santa Fe, NM

4. Where is Standardization Needed?
Performance data
Trace records vs. summary statistics
Data format
Data semantics
Library interfaces
Access to hardware counters
Statistical profiling
Dynamic instrumentation
October 18, 2001
LACSI Symposium, Santa Fe, NM

5. Standardization? (cont.)
User interfaces
Common set of commands
Common functionality
Timing routines
Memory utilization information
October 18, 2001
LACSI Symposium, Santa Fe, NM

6. Parallel Tools Consortium
Interaction between vendors, researchers, and users
Venue for standardization
Current projects
PAPI
DPCL
October 18, 2001
LACSI Symposium, Santa Fe, NM

7. LACSI Symposium, Santa Fe, NM
Hardware Counters
Small set of registers that count events, which are occurrences of specific signals related to the processor’s function
Monitoring these events facilitates correlation between the structure of the source/object code and the efficiency of the mapping of that code to the underlying architecture.
October 18, 2001
LACSI Symposium, Santa Fe, NM

8. LACSI Symposium, Santa Fe, NM
Goals of PAPI
Solid foundation for cross platform performance analysis tools
Free tool developers from re-implementing counter access
Standardization between vendors, academics and users
Encourage vendors to provide hardware and OS support for counter access
Reference implementations for a number of HPC architectures
Well documented and easy to use
October 18, 2001
LACSI Symposium, Santa Fe, NM

9. PAPI Implementation Tools!!! PAPI High Level PAPI Low Level Portable
Hardware Performance Counter
Operating System
Kernel Extension
PAPI Machine
Dependent Substrate
Machine
Specific
Layer
Portable
October 18, 2001
LACSI Symposium, Santa Fe, NM

10. LACSI Symposium, Santa Fe, NM
PAPI Preset Events
Proposed standard set of events deemed most relevant for application performance tuning
Defined in papiStdEventDefs.h
Mapped to native events on a given platform
Run tests/avail to see list of PAPI preset events available on a platform
October 18, 2001
LACSI Symposium, Santa Fe, NM

11. Statistical Profiling
PAPI provides support for execution profiling based on any counter event.
PAPI_profil() creates a histogram by text address of overflow counts for a specified region of the application code.
Used in vprof tool from Sandia Lab
October 18, 2001
LACSI Symposium, Santa Fe, NM

12. PAPI Reference Implementations
Linux/x86, Windows 2000
Requires patch to Linux kernel, driver for Windows
Linux/IA-64
Sun Solaris 2.8/Ultra I/II
IBM AIX 4.3+/Power
Contact IBM for pmtoolkit
SGI IRIX/MIPS
Compaq Tru64/Alpha Ev6 & Ev67
Requires OS device driver patch from Compaq
Per-thread and per-process counts not possible
Extremely limited number of events
Cray T3E/Unicos
October 18, 2001
LACSI Symposium, Santa Fe, NM

13. LACSI Symposium, Santa Fe, NM
PAPI Future Work
Improve accuracy of hardware counter and statistical profiling data
Microbenchmarks to measure accuracy (Pat Teller, UTEP)
Use hardware support for overflow interrupts
Use Event Address Registers (EARs) where available
Data structure based performance counters (collaboration with UMd)
Qualify event counting by address range
Page level counters in cache coherence hardware
October 18, 2001
LACSI Symposium, Santa Fe, NM

14. LACSI Symposium, Santa Fe, NM
PAPI Future (cont.)
Memory utilization extensions (following list suggested by Jack Horner, LANL)
Memory available on a node
Total memory available/used
High-water-mark memory used by process/thread
Disk swapping by process
Process-memory locality
Location of memory used by an object
Dynamic instrumentation – e.g., PAPI probe modules
October 18, 2001
LACSI Symposium, Santa Fe, NM

15. LACSI Symposium, Santa Fe, NM
For More Information
Software and documentation
Reference materials
Papers and presentations
Third-party tools
Mailing lists
October 18, 2001
LACSI Symposium, Santa Fe, NM

16. LACSI Symposium, Santa Fe, NM
DPCL
Dynamic Probe Class Library
Built of top of IBM version of University of Maryland’s dyninst
Current platforms
IBM AIX
Linux/x86 (limited functionality)
Dyninst ported to more platforms but by itself lacks functionality for easily instrumenting parallel applications.
October 18, 2001
LACSI Symposium, Santa Fe, NM

17. Infrastructure Components?
Parsers for common languages
Access to hardware counter data
Communication behavior instrumentation and analysis
Dynamic instrumentation capability
Runtime control of data collection and analysis
Performance data management
October 18, 2001
LACSI Symposium, Santa Fe, NM

18. LACSI Symposium, Santa Fe, NM
Case Studies
Test tools on large-scale applications in production environment
Reveal limitations of tools and point out areas where improvements are needed
Develop performance tuning methodologies for large-scale codes
October 18, 2001
LACSI Symposium, Santa Fe, NM

19. PERC: Performance Evaluation Research Center
Developing a science for understanding performance of scientific applications on high-end computer systems.
Developing engineering strategies for improving performance on these systems.
DOE Labs: ANL, LBNL, LLNL, ORNL
Universities: UCSD, UI-UC, UMD, UTK
Funded by SciDAC: Scientific Discovery through Advanced Computing
October 18, 2001
LACSI Symposium, Santa Fe, NM

20. PERC: Real-World Applications
High Energy and Nuclear Physics
Shedding New Light on Exploding Stars: Terascale Simulations of Neutrino-Driven SuperNovae and Their NucleoSynthesis
Advanced Computing for 21st Century Accelerator Science and Technology
Biology and Environmental Research
Collaborative Design and Development of the Community Climate System Model for Terascale Computers
Fusion Energy Sciences
Numerical Computation of Wave-Plasma Interactions in Multi-dimensional Systems
Advanced Scientific Computing
Terascale Optimal PDE Solvers (TOPS)
Applied Partial Differential Equations Center (APDEC)
Scientific Data Management (SDM)
Chemical Sciences
Accurate Properties for Open-Shell States of Large Molecules
…and more…
October 18, 2001
LACSI Symposium, Santa Fe, NM

21. Parallel Climate Transition Model
Components for Ocean, Atmosphere, Sea Ice, Land Surface and River Transport
Developed by Warren Washington’s group at NCAR
POP: Parallel Ocean Program from LANL
CCM3: Community Climate Model 3.2 from NCAR including LSM: Land Surface Model
ICE: CICE from LANL and CCSM from NCAR
RTM: River Transport Module from UT Austin
Fortran 90 with MPI
October 18, 2001
LACSI Symposium, Santa Fe, NM

22. PCTM: Parallel Climate Transition Model
River Model
Ocean Model
Atmosphere Model
Flux Coupler
Land Surface Model
Sea Ice
Model
Sequential Execution of Parallelized Modules
October 18, 2001
LACSI Symposium, Santa Fe, NM

23. LACSI Symposium, Santa Fe, NM
PCTM Instrumentation
Vampir tracefile in tens of gigabytes range even for toy problem
Hand instrumentation with PAPI tedious
UIUC working on SvPablo instrumentation
Must work in batch queueing environment
Plan to try other tools
MPE logging and jumpshot
TAU
VGV?
October 18, 2001
LACSI Symposium, Santa Fe, NM

24. LACSI Symposium, Santa Fe, NM
In Progress
Standardization and reference implementations for memory utilization information (funded by DoD HPCMP PET, Ptools-sponsored project)
Repositories of application performance evaluation case studies (e.g., SciDAC PERC)
Portable dynamic instrumentation for parallel applications (DOE MICS project – UTK, UMd, UWisc)
Increased functionality and accuracy of hardware counter data collection (DoD HPCMP, DOE MICS)
October 18, 2001
LACSI Symposium, Santa Fe, NM

25. LACSI Symposium, Santa Fe, NM
Next Steps
Additional areas for standardization?
Scalable trace file format
Metadata standards for performance data
New hardware counter metrics (e.g., SMP and DMP events, data-centric counters)
Others?
October 18, 2001
LACSI Symposium, Santa Fe, NM

26. LACSI Symposium, Santa Fe, NM
Next Steps (cont.)
Sharing of tools and data
Open source software
Machine and software profiles
Runtime performance data
Benchmark results
Application examples and case studies
Long-term goal: common performance tool infrastructure across HPC systems
October 18, 2001
LACSI Symposium, Santa Fe, NM