1. Allen D. Malony malony@cs.uoregon.edu Department of Computer and Information Science Computational Science Institute University of Oregon Integrating Performance Analysis in Complex Scientific Software: Experiences with the Uintah Computational Framework

2. Research Centre Juelich April 9, 2002 Acknowledgements  Sameer Shende, Robert Bell University of Oregon  Steven Parker, J. Dav de St.-Germain, and Alan Morris University of Utah  Department of Energy (DOE), ASCI Academic Strategic Alliances Program (ASAP)  Center for Simulation of Accidental Fires and Explosions (C-SAFE), ASCI/ASAP Level 1 center, University of Utah, http://www.csafe.utah.eduhttp://www.csafe.utah.edu  Computational Science Institute, ASCI/ASAP Level 3 projects with LLNL / LANL, University of Oregon, http://www.csi.uoregon.eduhttp://www.csi.uoregon.edu

3. Research Centre Juelich April 9, 2002 Complex Parallel Systems  Complexity in computing system architecture  Diverse parallel system architectures  shared / distributed memory, cluster, hybrid, NOW, Grid, …  Sophisticated processor and memory architectures  Advanced network interface and switching architecture  Specialization of hardware components  Complexity in parallel software environment  Diverse parallel programming paradigms  shared memory multi-threading, message passing, hybrid  Hierarchical, multi-level software architectures  Optimizing compilers and sophisticated runtime systems  Advanced numerical libraries and application frameworks

4. Research Centre Juelich April 9, 2002 Complexity Drives Performance Need / Technology  Observe/analyze/understand performance behavior  Multiple levels of software and hardware  Different types and detail of performance data  Alternative performance problem solving methods  Multiple targets of software and system application  Robust AND ubiquitous performance technology  Broad scope of performance observability  Flexible and configurable mechanisms  Technology integration and extension  Cross-platform portability  Open, layered, and modular framework architecture

5. Research Centre Juelich April 9, 2002 What is Parallel Performance Technology?  Performance instrumentation tools  Different program code levels  Different system levels  Performance measurement (observation) tools  Profiling and tracing of SW/HW performance events  Different software (SW) and hardware (HW) levels  Performance analysis tools  Performance data analysis and presentation  Online and offline tools  Performance experimentation and data management  Performance modeling and prediction tools

6. Research Centre Juelich April 9, 2002 Complexity Challenges for Performance Tools  Computing system environment complexity  Observation integration and optimization  Access, accuracy, and granularity constraints  Diverse/specialized observation capabilities/technology  Restricted modes limit performance problem solving  Sophisticated software development environments  Programming paradigms and performance models  Performance data mapping to software abstractions  Uniformity of performance abstraction across platforms  Rich observation capabilities and flexible configuration  Common performance problem solving methods

7. Research Centre Juelich April 9, 2002 General Problems How do we create robust and ubiquitous performance technology for the analysis and tuning of parallel and distributed software and systems in the presence of (evolving) complexity challenges? How do we apply performance technology effectively for the variety and diversity of performance problems that arise in the context of complex parallel and distributed computer systems? 

8. Research Centre Juelich April 9, 2002 Scientific Software Engineering  Modern scientific simulation software is complex  Large development teams of diverse expertise  Simultaneous development on different system parts  Iterative, multi-stage, long-term software development  Need support for managing complex software process  Software engineering tools for revision control, automated testing, and bug tracking are commonplace  Tools for HPC performance engineering are not  evaluation (measurement, analysis, benchmarking)  optimization (diagnosis, tracking, prediction, tuning)  Incorporate performance engineering methodology and support by flexible and robust performance tools

9. Research Centre Juelich April 9, 2002 Computation Model for Performance Technology  How to address dual performance technology goals?  Robust capabilities + widely available methodologies  Contend with problems of system diversity  Flexible tool composition/configuration/integration  Approaches  Restrict computation types / performance problems  limited performance technology coverage  Base technology on abstract computation model  general architecture and software execution features  map features/methods to existing complex system types  develop capabilities that can adapt and be optimized

10. Research Centre Juelich April 9, 2002 General Complex System Computation Model  Node: physically distinct shared memory machine  Message passing node interconnection network  Context: distinct virtual memory space within node  Thread: execution threads (user/system) in context memory Node VM space Context SMP Threads node memory … … Interconnection Network Inter-node message communication * * physical view model view

11. Research Centre Juelich April 9, 2002 Framework for Performance Problem Solving  Model-based performance technology  Instrumentation / measurement / execution models  performance observability constraints  performance data types and events  Analysis / presentation model  performance data processing  performance views and model mapping  Integration model  performance tool component configuration / integration  Can a performance problem solving framework be designed based on a general complex system model and with a performance technology model approach?

12. Research Centre Juelich April 9, 2002 TAU Performance System Framework  Tuning and Analysis Utilities  Performance system framework for scalable parallel and distributed high-performance computing  Targets a general complex system computation model  nodes / contexts / threads  Multi-level: system / software / parallelism  Measurement and analysis abstraction  Integrated toolkit for performance instrumentation, measurement, analysis, and visualization  Portable performance profiling/tracing facility  Open software approach

13. Research Centre Juelich April 9, 2002 TAU Performance System Architecture EPILOG Paraver

14. Research Centre Juelich April 9, 2002 Pprof Output (NAS Parallel Benchmark – LU)  Intel Quad PIII Xeon, RedHat, PGI F90  F90 + MPICH  Profile for: Node Context Thread  Application events and MPI events

15. Research Centre Juelich April 9, 2002 jRacy (NAS Parallel Benchmark – LU) n: node c: context t: thread Global profiles Individual profile Routine profile across all nodes

16. Research Centre Juelich April 9, 2002 TAU + PAPI (NAS Parallel Benchmark – LU )  Floating point operations  Replaces execution time  Only requires re-linking to different TAU library

17. Research Centre Juelich April 9, 2002 TAU + Vampir (NAS Parallel Benchmark – LU) Timeline display Communications display Parallelism display Callgraph display

18. Research Centre Juelich April 9, 2002 Utah ASCI/ASAP Level 1 Center (C-SAFE)  C-SAFE was established to build a problem-solving environment (PSE) for the numerical simulation of accidental fires and explosions  Fundamental chemistry and engineering physics models  Coupled with non-linear solvers, optimization, computational steering, visualization, and experimental data verification  Very large-scale simulations  Computer science problems:  Coupling of multiple simulation codes  Software engineering across diverse expert teams  Achieving high performance on large-scale systems

19. Research Centre Juelich April 9, 2002 Example C-SAFE Simulation Problems ∑ Heptane fire simulation Material stress simulation Typical C-SAFE simulation with a billion degrees of freedom and non-linear time dynamics

20. Research Centre Juelich April 9, 2002 Uintah Problem Solving Environment  Enhanced SCIRun PSE  Pure dataflow to component-based  Shared memory to scalable multi-/mixed-mode parallelism  Interactive only to interactive and standalone  Design and implement Uintah component architecture  Application programmers provide  description of computation (tasks and variables)  code to perform task on single “patch” (sub-region of space)  Follow Common Component Architecture (CCA) model  Design and implement Uintah Computational Framework (UCF) on top of the component architecture

21. Research Centre Juelich April 9, 2002 Uintah High-Level Component View

22. Research Centre Juelich April 9, 2002 High Level Architecture C-SAFE Implicitly Connected to All Components UCF Data Control / Light Data Checkpointing Mixing Model Mixing Model Fluid Model Fluid Model Subgrid Model Subgrid Model Chemistry Database Controller Chemistry Database Controller Chemistry Databases Chemistry Databases High Energy Simulations High Energy Simulations Numerical Solvers Numerical Solvers Non-PSE Components Performance Analysis Performance Analysis Simulation Controller Simulation Controller Problem Specification Numerical Solvers Numerical Solvers MPM Material Properties Database Material Properties Database Blazer Database Visualization Data Manager Data Manager Post Processing And Analysis Post Processing And Analysis Parallel Services Parallel Services Resource Management Resource Management PSE Components Scheduler Uintah Parallel Component Architecture

23. Research Centre Juelich April 9, 2002 Uintah Computational Framework  Execution model based on software (macro) dataflow  Exposes parallelism and hides data transport latency  Computations expressed a directed acyclic graphs of tasks  consumes input and produces output (input to future task)  input/outputs specified for each patch in a structured grid  Abstraction of global single-assignment memory  DataWarehouse  Directory mapping names to values (array structured)  Write value once then communicate to awaiting tasks  Task graph gets mapped to processing resources  Communications schedule approximates global optimal

24. Research Centre Juelich April 9, 2002 Uintah Task Graph (Material Point Method)  Diagram of named tasks (ovals) and data (edges)  Imminent computation  Dataflow-constrained  MPM  Newtonian material point motion time step  Solid: values defined at material point (particle)  Dashed: values defined at vertex (grid)  Prime (‘): values updated during time step

25. Research Centre Juelich April 9, 2002 Example Taskgraphs (MPM and Coupled)

26. Research Centre Juelich April 9, 2002 Taskgraph Advantages  Accommodates flexible integration needs  Accommodates a wide range of unforeseen work loads  Accommodates a mix of static and dynamic load balance  Manage complexity of mixed-mode programming  Avoids unnecessary transport abstraction overheads  Simulation time/space coupling  Allows uniform abstraction for coordinating coupled models’ time and grid scales  Allows application components and framework infrastructure (e.g., scheduler) to evolve independently

27. Research Centre Juelich April 9, 2002 Uintah PSE  UCF automatically sets up:  Domain decomposition  Inter-processor communication with aggregation/reduction  Parallel I/O  Checkpoint and restart  Performance measurement and analysis (stay tuned)  Software engineering  Coding standards  CVS (Commits: Y3 - 26.6 files/day, Y4 - 29.9 files/day)  Correctness regression testing with bugzilla bug tracking  Nightly build (parallel compiles)  170,000 lines of code (Fortran and C++ tasks supported)

28. Research Centre Juelich April 9, 2002 Performance Technology Integration  Uintah present challenges to performance integration  Software diversity and structure  UCF middleware, simulation code modules  component-based hierarchy  Portability objectives  cross-language and cross-platform  multi-parallelism: thread, message passing, mixed  Scalability objectives  High-level programming and execution abstractions  Requires flexible and robust performance technology  Requires support for performance mapping

29. Research Centre Juelich April 9, 2002 Performance Analysis Objectives for Uintah  Micro tuning  Optimization of simulation code (task) kernels for maximum serial performance  Scalability tuning  Identification of parallel execution bottlenecks  overheads: scheduler, data warehouse, communication  load imbalance  Adjustment of task graph decomposition and scheduling  Performance tracking  Understand performance impacts of code modifications  Throughout course of software development  C-SAFE application and UCF software

30. Research Centre Juelich April 9, 2002 Uintah Performance Engineering Approach  Contemporary performance methodology focuses on control flow (function) level measurement and analysis  C-SAFE application involves coupled-models with task- based parallelism and dataflow control constraints  Performance engineering on algorithmic (task) basis  Observe performance based on algorithm (task) semantics  Analyze task performance characteristics in relation to other simulation tasks and UCF components  scientific component developers can concentrate on performance improvement at algorithmic level  UCF developers can concentrate on bottlenecks not directly associated with simulation module code

31. Research Centre Juelich April 9, 2002 Task execution time dominates (what task?) MPI communication overheads (where?) Task Execution in Uintah Parallel Scheduler  Profile methods and functions in scheduler and in MPI library Task execution time distribution  Need to map performance data!

32. Research Centre Juelich April 9, 2002 Semantics-Based Performance Mapping  Associate performance measurements with high-level semantic abstractions  Need mapping support in the performance measurement system to assign data correctly

33. Research Centre Juelich April 9, 2002 Hypothetical Mapping Example  Particles distributed on surfaces of a cube Particle* P[MAX]; /* Array of particles */ int GenerateParticles() { /* distribute particles over all faces of the cube */ for (int face=0, last=0; face < 6; face++){ /* particles on this face */ int particles_on_this_face = num(face); for (int i=last; i < particles_on_this_face; i++) { /* particle properties are a function of face */ P[i] =... f(face);... } last+= particles_on_this_face; }

34. Research Centre Juelich April 9, 2002 Hypothetical Mapping Example (continued)  How much time is spent processing face i particles?  What is the distribution of performance among faces?  How is this determined if execution is parallel? int ProcessParticle(Particle *p) { /* perform some computation on p */ } int main() { GenerateParticles(); /* create a list of particles */ for (int i = 0; i < N; i++) /* iterates over the list */ ProcessParticle(P[i]); }

35. Research Centre Juelich April 9, 2002 Semantic Entities/Attributes/Associations (SEAA)  New dynamic mapping scheme (S. Shende, Ph.D. thesis)  Contrast with ParaMap (Miller and Irvin)  Entities defined at any level of abstraction  Attribute entity with semantic information  Entity-to-entity associations  Two association types (implemented in TAU API)  Embedded – extends data structure of associated object to store performance measurement entity  External – creates an external look-up table using address of object as the key to locate performance measurement entity

36. Research Centre Juelich April 9, 2002 No Performance Mapping versus Mapping  Typical performance tools report performance with respect to routines  Does not provide support for mapping  Performance tools with SEAA mapping can observe performance with respect to scientist’s programming and problem abstractions TAU (no mapping) TAU (w/ mapping)

37. Research Centre Juelich April 9, 2002 Uintah Task Performance Mapping  Uintah partitions individual particles across processing elements (processes or threads)  Simulation tasks in task graph work on particles  Tasks have domain-specific character in the computation  “interpolate particles to grid” in Material Point Method  Task instances generated for each partitioned particle set  Execution scheduled with respect to task dependencies  How to attributed execution time among different tasks  Assign semantic name (task type) to a task instance  SerialMPM::interpolateParticleToGrid  Map TAU timer object to (abstract) task (semantic entity)  Look up timer object using task type (semantic attribute)  Further partition along different domain-specific axes

38. Research Centre Juelich April 9, 2002 Task Performance Mapping Instrumentation void MPIScheduler::execute(const ProcessorGroup * pc, DataWarehouseP & old_dw, DataWarehouseP & dw ) {... TAU_MAPPING_CREATE( task->getName(), "[MPIScheduler::execute()]", (TauGroup_t)(void*)task->getName(), task->getName(), 0);... TAU_MAPPING_OBJECT(tautimer) TAU_MAPPING_LINK(tautimer,(TauGroup_t)(void*)task->getName()); // EXTERNAL ASSOCIATION... TAU_MAPPING_PROFILE_TIMER(doitprofiler, tautimer, 0) TAU_MAPPING_PROFILE_START(doitprofiler,0); task->doit(pc); TAU_MAPPING_PROFILE_STOP(0);... }

39. Research Centre Juelich April 9, 2002 Task Performance Mapping (Profile) Performance mapping for different tasks Mapped task performance across processes

40. Research Centre Juelich April 9, 2002 Task Performance Mapping (Trace) Work packet computation events colored by task type Distinct phases of computation can be identifed based on task

41. Research Centre Juelich April 9, 2002 Task Performance Mapping (Trace - Zoom) Startup communication imbalance

42. Research Centre Juelich April 9, 2002 Task Performance Mapping (Trace - Parallelism) Communication / load imbalance

43. Research Centre Juelich April 9, 2002 Comparing Uintah Traces for Scalability Analysis 8 processes 32 processes

44. Research Centre Juelich April 9, 2002 Scaling Performance Optimizations Last year: initial “correct” scheduler Reduce communication by 10 x Reduce task graph overhead by 20 x ASCI Nirvana SGI Origin 2000 Los Alamos National Laboratory

45. Research Centre Juelich April 9, 2002 Scalability to 2000 Processors (Fall 2001) ASCI Nirvana SGI Origin 2000 Los Alamos National Laboratory

46. Research Centre Juelich April 9, 2002 Performance Tracking and Reporting  Integrated performance measurement allows performance analysis throughout development lifetime  Applied performance engineering in software design and development (software engineering) process  Create “performance portfolio” from regular performance experimentation (coupled with software testing)  Use performance knowledge in making key software design decision, prior to major development stages  Use performance benchmarking and regression testing to identify irregularities  Support automatic reporting of performance bugs  Cross-platform (cross-generation) evaluation

47. Research Centre Juelich April 9, 2002 XPARE - eXPeriment Alerting and REporting  Experiment launcher automates measurement / analysis  Configuration and compilation of performance tools  Uintah instrumentation control for experiment type  Multiple experiment execution  Performance data collection, analysis, and storage  Integrated in Uintah software testing harness  Reporting system conducts performance regression tests  Apply performance difference thresholds (alert ruleset)  Alerts users via email if thresholds have been exceeded  Web alerting setup and full performance data reporting  Historical performance data analysis

48. Research Centre Juelich April 9, 2002 XPARE System Architecture Experiment Launch Mail server Performance Database Performance Reporter Comparison Tool Regression Analyzer Alerting Setup Web server

49. Research Centre Juelich April 9, 2002 Alerting Setup

50. Research Centre Juelich April 9, 2002 Experiment Results Viewing Selection

51. Research Centre Juelich April 9, 2002 Web-Based Experiment Reporting

52. Research Centre Juelich April 9, 2002 Web-Based Experiment Reporting (continued)

53. Research Centre Juelich April 9, 2002 Web-Based Experiment Reporting (continued)

54. Research Centre Juelich April 9, 2002 Performance Analysis Tool Integration  Complex systems pose challenging performance analysis problems that require robust methodologies and tools  New performance problems will arise  Instrumentation and measurement  Data analysis and presentation  Diagnosis and tuning  No one performance tool can address all concerns  Look towards an integration of performance technologies  Support to link technologies to create performance problem solving environments  Performance engineering methodology and tool integration with software design and development process

55. Research Centre Juelich April 9, 2002 Integrated Performance Evaluation Environment

56. Research Centre Juelich April 9, 2002 References  A. Malony and S. Shende, “Performance Technology for Complex Parallel and Distributed Systems,” Proc. 3rd Workshop on Parallel and Distributed Systems (DAPSYS), pp. 37-46, Aug. 2000.  S. Shende, A. Malony, and R. Ansell-Bell, “Instrumentation and Measurement Strategies for Flexible and Portable Empirical Performance Evaluation,” Proc. Int’l. Conf. on Parallel and Distributed Processing Techniques and Applications (PDPTA), CSREA, pp. 1150-1156, July 2001.  S. Shende, “The Role of Instrumentation and Mapping in Performance Measurement,” Ph.D. Dissertation, Univ. of Oregon, Aug. 2001.  J. de St. Germain, A. Morris, S. Parker, A. Malony, and S. Shende, “Integrating Performance Analysis in the Uintah Software Development Cycle,” ISHPC 2002, Nara, Japan, May, 2002.