1. Challenges in Performance Evaluation and Improvement of Scientific Codes
Boyana Norris
Argonne National Laboratory
Ivana Veljkovic
Pennsylvania State University

2. Outline Performance evaluation challenges Component-based approach
Motivating example: adaptive linear system solution
A component infrastructure for performance monitoring and adaptation of applications
Summary and future work
February 13, 2005
SIAM CSE

3. Acknowledgments Ivana Veljkovic, Padma Raghavan (Penn State)
Sanjukta Bhowmick (ANL/Columbia)
Lois Curfman McInnes (ANL)
TAU developers (U. Oregon)
PERC members
Sponsor: DOE and NSF
February 13, 2005
SIAM CSE

4. Challenges in performance evaluation
Many tools for performance data gathering and analysis
PAPI, TAU, SvPablo, Kojak, …
Various interfaces, levels of automation, and approaches to information presentation
User’s point of view
What do the different tools do? Which is most appropriate for a given application?
(How) can multiple tools be used in concert?
I have tons of performance data, now what?
What automatic tuning tools are available, what exactly do they do?
How hard is it to install/learn/use tool X?
Is instrumented code portable? What’s the overhead of instrumentation? How does code evolution affect the performance analysis process?
February 13, 2005
SIAM CSE

5. Incomplete list of tools
Source instrumentation: TAU/PDT, KOJAK (MPI/OpenMP), SvPablo, Performance Assertions, …
Binary instrumentation: HPCToolkit, Paradyn, DyninstAPI, …
Performance monitoring: MetaSim Tracer (memory), PAPI, HPCToolkit, Sigma++ (memory), DPOMP (OpenMP), mpiP, gprof, psrun, …
Modeling/analysis/prediction: MetaSim Convolver (memory), DIMEMAS(network), SvPablo (scalability), Paradyn, Sigma++, …
Source/binary optimization: Automated Empirical Optimization of Software (ATLAS), OSKI, ROSE
Runtime adaptation: ActiveHarmony, SALSA
PAPI: hardware counters, HPCToolkit: statistical sampling of hw counters
End-to-end tools: POEMS – not available?
– no soft. download
February 13, 2005
SIAM CSE

6. Incomplete list of tools
Source instrumentation: TAU/PDT, KOJAK (MPI/OpenMP), SvPablo, Performance Assertions, …
Binary instrumentation: HPCToolkit, Paradyn, DyninstAPI, …
Performance monitoring: MetaSim Tracer (memory), PAPI, HPCToolkit, Sigma++ (memory), DPOMP (OpenMP), mpiP, gprof, psrun, …
Modeling/analysis/prediction: MetaSim Convolver (memory), DIMEMAS(network), SvPablo (scalability), Paradyn, Sigma++, …
Source/binary optimization: Automated Empirical Optimization of Software (ATLAS), OSKI, ROSE
Runtime adaptation: ActiveHarmony, SALSA
PAPI: hardware counters, HPCToolkit: statistical sampling of hw counters
End-to-end tools: POEMS – not available?
– no soft. download
February 13, 2005
SIAM CSE

7. Challenges (where is the complexity?)
More effective use  integration
Tool developer’s perspective
Overhead of initially implementing one-to-one interoperabilty
Managing dependencies on other tools
Maintaining interoperabilty as different tools evolve
Individual Scientist Perspective
Learning curve for performance tools  less time to focus on own research (modeling, physics, mathematics)
Potentially significant time investment needed to find out whether/how using someone else’s tool would improve performance  tend to do own hand-coded optimizations (time-consuming, non-reusable)
Lack of tools that automate (at least partially) algorithm discovery, assembly, configuration, and enable runtime adaptivity
February 13, 2005
SIAM CSE

8. What can be done How to manage complexity? Provide
Performance tools that are truly interoperable
Uniform easy access to tools
Component implementations of software, esp. supporting numerical codes, such as linear algebra algorithms
New algorithms (e.g., interactive/dynamic techniques, algorithm composition)
Implementation approach: components, both for tools and the application software
February 13, 2005
SIAM CSE

9. What is being done
No “integrated” environment for performance monitoring, analysis, and optimization
Most past efforts
One-to-one tool interoperability
More recently
OSPAT (initial meeting at SC’04), focus on common data representation and interfaces
Tool-independent performance databases: PerfDMF
Eclipse parallel tools project (LANL) …
February 13, 2005
SIAM CSE

10. OSPAT The following areas were recommended for OSPAT to investigate:
A common instrumentation API for source level, compiler level, library level, binary instrumentation
A common probe interface for routine entry and exit events
A common profile database schema
An API to walk the callstack and examine the heap memory
A common API for thread creation and fork interface
Visualization components for drawing histograms and hierarchical displays typically used by performance tools
February 13, 2005
SIAM CSE

11. Components
Working definition: a component is a piece of software that can be composed with other components within a framework; composition can be either static (at link time) or dynamic (at run time)
“plug-and-play” model for building applications
For more info: C. Szyperski, Component Software: Beyond Object-Oriented Programming, ACM Press, New York, 1998
Components enable
Tool interoperability
Automation of performance instrumentation/monitoring
Application adaptivity (automated or user-guided)
February 13, 2005
SIAM CSE

12. Example: component infrastructure for multimethod linear solvers
Goal: provide a framework for
Performance monitoring of numerical components
Dynamic adaptativity, based on:
Off-line analyses of past performance information
Online analysis of current execution performance information
Motivating application examples:
Driven cavity flow [Coffey et al, 2003], nonlinear PDE solution
FUN3D – incompressible and compressible Euler equations
Prior work in multimethod linear solvers
McInnes et al, ’03, Bhowmick et al,’03 and ’05, Norris at al. ’05.
February 13, 2005
SIAM CSE

13. Example: driven cavity flow
Linear solver: GMRES(30), vary only fill level of ILU preconditioner
Adaptive heuristic based on:
Previous linear solution convergence rate, nonlinear solution convergence rate, rate of increase of linear solution iterations
96x96 mesh, Grashof = 105, lid velocity = 100
Intel P4 Xeon, dual 2.2 GHz, 4GB RAM
February 13, 2005
SIAM CSE

14. Example: Compressible PETSc-FUN3D
Finite volume discretization, variable order Roe scheme on a tetrahedral, vertex-centered mesh
Initial discretization: first-order scheme; switch to second-order after shock position has settled down
Large sparse linear system solution takes approximately 72% of overall solution time
Original FUN3D developer: W.K. Anderson et al., NASA Langley
Image: Dinesh Kaushik
February 13, 2005
SIAM CSE

15. PETSc-FUN3d, cont.
A3: Nonsequence-based adaptive strategy based on polynomial interpolation [Bhowmick et al., ’05]
A3 vs base method time: ~1% slowdown - 32% improvement
Hand-tuned adaptive vs base method time: 7% - 42% improvement
February 13, 2005
SIAM CSE

16. Component architecture
Off-line analysis
PerfDMF
Runtime DB
extract
extract
insert
Metadata extractor
Checkpoint
TAU
query
extract
Monitor
checkpoint
adapt request
start, stop, trigger
adapt: algorithm, parameters
Experiment
February 13, 2005
SIAM CSE

17. Future work Integration of ongoing efforts in Long term
Performance tools: common interfaces and data represenation (leverage OSPAT, PerfDMF, TAU performance interfaces, and similar efforts)
Numerical components: emerging common interfaces (e.g., TOPS solver interfaces) increase choice of solution method  automated composition and adaptation strategies
Long term
Is a more organized (but not too restrictive) environment for scientific software lifecycle development possible/desirable?
February 13, 2005
SIAM CSE

18. Typical application development “cycle”
Configure, make,…
Compilation,
Linking
Ext. dependencies,
Version control
Debugging
Implementation
Testing
Performance evaluation
Deployment
Design
Performance tools
Production
Execution
Job management,
Results
February 13, 2005
SIAM CSE

19. Future work Beyond components Work flow
Reproducible results – associate all necessary information for reproducing particular application instance
Ontology of tools and tools to guide selection and use
February 13, 2005
SIAM CSE

20. Summary
No shortage of performance evaluation, analysis, and optimization technology (and new capabilities are continuously added)
Little shared infrastructure, limiting the utility of performance technology in scientific computing
Components, both in performance tools, and numerical software can be used to manage complexity and enable better performance through dynamic adaptation or multimethod solvers
A life-cycle environment may be the best long-term solution
Some relevant sites:
(performance tools)
(component specification)
February 13, 2005
SIAM CSE