1. Allen D. Malony, Sameer Shende {malony,sameer}@cs.uoregon.edu Department of Computer and Information Science Computational Science Institute University of Oregon Performance Technology for Parallel Component Software

2. University of Vienna 2 Jan. 22, 2003 Outline  Overview of the TAU project  Performance analysis of distributed Java programs  Profiling toolkit, PDT, Database, Monitor, XPARE  Performance Engineered Component Software  CCA Performance Observation Component  CCAFFEINE (Classic C++)  SIDL  Applications :  Optimizer Component  Combustion Component  Concluding remarks

3. University of Vienna 3 Jan. 22, 2003 TAU Performance System Framework  Tuning and Analysis Utilities  Performance system framework for scalable parallel and distributed high-performance computing  Integrated toolkit for performance instrumentation, measurement, analysis, and visualization  Portable, configurable performance profiling/tracing facility  Open software approach  University of Oregon, LANL, FZJ Germany  http://www.cs.uoregon.edu/research/paracomp/tau http://www.cs.uoregon.edu/research/paracomp/tau

4. University of Vienna 4 Jan. 22, 2003 General Complex System Computation Model  Node: physically distinct shared memory machine  Message passing node interconnection network  Context: distinct virtual memory space within node  Thread: execution threads (user/system) in context memory Node VM space Context SMP Threads node memory … … Interconnection Network Inter-node message communication * * physical view model view

5. University of Vienna 5 Jan. 22, 2003 TAU Performance System Architecture EPILOG Paraver

6. University of Vienna 6 Jan. 22, 2003 TAU Instrumentation  Flexible instrumentation mechanisms at multiple levels  Source code  manual  automatic using Program Database Toolkit (PDT), OPARI  Object code  pre-instrumented libraries (e.g., MPI using PMPI)  statically linked  dynamically linked (e.g., Virtual machine instrumentation)  fast breakpoints (compiler generated)  Executable code  dynamic instrumentation (pre-execution) using DynInstAPI

7. University of Vienna 7 Jan. 22, 2003 Virtual Machine Performance Instrumentation  Integrate performance system with VM  Captures robust performance data (e.g., thread events)  Maintain features of environment  portability, concurrency, extensibility, interoperation  Allow use in optimization methods  JVM Profiling Interface (JVMPI)  Generation of JVM events and hooks into JVM  Profiler agent (TAU) loaded as shared object  registers events of interest and address of callback routine  Access to information on dynamically loaded classes  No need to modify Java source, bytecode, or JVM

8. University of Vienna 8 Jan. 22, 2003 TAU Profiling of Java Application (SciVis) Profile for each Java thread Captures events for different Java packages 24 threads of execution!

9. University of Vienna 9 Jan. 22, 2003 Mixed-mode Parallel Programs (Java + MPI)  Java threads and MPI communications  Shared-memory multi-threading events  Message communications events  Unified performance measurement and views  Integration of performance mechanisms  Integrated association of performance events  thread event and communication events  user-defined (source-level) performance events  JVM events  Support for performance measurement scaling  Support for performance data access

10. University of Vienna 10 Jan. 22, 2003 Instrumentation and Measurement Cooperation  Problem  JVMPI doesn’t see MPI events (e.g., rank (node))  MPI profiling interfaces doesn’t see threads  Source instrumentation doesn’t see either!  Need cooperation between interfaces  MPI exposes rank, gets thread information  JVMPI exposes thread information, get rank  Source instrumentation gets both  Post-mortem matching of sends and receives  Selective instrumentation  java -XrunTAU:exclude=java/io,sun

11. University of Vienna 11 Jan. 22, 2003 JVMPI Thread API Event notification TAU Java Instrumentation Architecture Java program TAU package mpiJava package MPI profiling interface TAU wrapper Native MPI library Profile DB JNI TAU

12. University of Vienna 12 Jan. 22, 2003 Java Source-Level Instrumentation  TAU Java package  User-defined events  TAU.Profile class for new “timers”  Start/Stop  Performance data output at end

13. University of Vienna 13 Jan. 22, 2003 Parallel Java Game of Life (Profile)  mpiJava testcase  4 nodes, 28 threads Node 0 Node 1 Node 2 Thread 4 executes all MPI routines Merged Java and MPI event profiles

14. University of Vienna 14 Jan. 22, 2003 Parallel Java Game of Life (Trace)  Integrated event tracing  Merged trace viz  Node process grouping  Thread message pairing  Vampir display  Multi-level event grouping

15. University of Vienna 15 Jan. 22, 2003 TAU Status  Instrumentation supported:  Source, preprocessor, compiler, MPI, runtime, virtual machine  Languages supported:  C++, C, F90, Java, Python  HPF, ZPL, HPC++, pC++...  Packages supported:  PAPI [UTK], PCL [FZJ] (hardware performance counter access),  Opari, PDT [UO,LANL,FZJ], DyninstAPI [U.Maryland] (instrumentation),  EXPERT, EPILOG[FZJ],Vampir[Pallas], Paraver [CEPBA] (visualization)  Platforms supported:  IBM SP, SGI Origin, Sun, HP Superdome, HP-Compaq ES,  Linux clusters (IA-32, IA-64, PowerPC, Alpha), Apple OS X, Windows,  Hitachi SR8000, NEC SX, Cray T3E...  Compilers suites supported:  GNU, Intel KAI (KCC, KAP/Pro), Intel, SGI, IBM, Compaq,HP, Fujitsu, Hitachi, Sun, Apple, Microsoft, NEC, Cray, PGI, Absoft, …  Thread libraries supported:  Pthreads, SGI sproc, OpenMP, Windows, Java, SMARTS

16. University of Vienna 16 Jan. 22, 2003 TAU Measurement System Configuration  configure [OPTIONS]  {-c++=, -cc= } Specify C++ and C compilers  {-pthread, -sproc}Use pthread or SGI sproc threads  -openmpUse OpenMP threads  -opari= Specify location of Opari OpenMP tool  -papi= Specify location of PAPI  -pdt= Specify location of PDT  -dyninst= Specify location of DynInst Package  {-mpiinc=, mpilib= }Specify MPI library instrumentation  - TRACE Generate TAU event traces  -PROFILE Generate TAU profiles  -MULTIPLECOUNTERSUse more than one hardware counter  -PROFILECALLPATHUse 1-level callpath profiles  -PAPIWALLCLOCKUse PAPI to access wallclock time  -PAPIVIRTUALUse PAPI for virtual (user) time …

17. University of Vienna 17 Jan. 22, 2003 TAU Measurement Configuration – Examples ./configure -c++=xlC -cc=xlc –pdt=/usr/packages/pdtoolkit-2.1 -pthread  Use TAU with IBM’s xlC compiler, PDT and the pthread library  Enable TAU profiling (default) ./configure -TRACE –PROFILE  Enable both TAU profiling and tracing ./configure -c++=guidec++ -cc=guidec -papi=/usr/local/packages/papi –openmp -mpiinc=/usr/packages/mpich/include -mpilib=/usr/packages/mpich/lib  Use OpenMP+MPI using KAI's Guide compiler suite and use PAPI for accessing hardware performance counters for measurements  Typically configure multiple measurement libraries

18. University of Vienna 18 Jan. 22, 2003 Program Database Toolkit (PDT)  Program code analysis framework for developing source- based tools  High-level interface to source code information  Integrated toolkit for source code parsing, database creation, and database query  commercial grade front end parsers  portable IL analyzer, database format, and access API  open software approach for tool development  Target and integrate multiple source languages  Use in TAU to build automated performance instrumentation tools

19. University of Vienna 19 Jan. 22, 2003 Program Database Toolkit Application / Library C / C++ parser Fortran 77/90 parser C / C++ IL analyzer Fortran 77/90 IL analyzer Program Database Files IL DUCTAPE PDBhtml SILOON CHASM TAU_instr Program documentation Application component glue C++ / F90 interoperability Automatic source instrumentation

20. University of Vienna 20 Jan. 22, 2003 Program Database Toolkit (PDT)  Program code analysis framework for developing source-based tools for C99, C++ and F90  High-level interface to source code information  Widely portable:  IBM, SGI, Compaq, HP, Sun, Linux clusters,Windows, Apple, Hitachi, Cray T3E...  Integrated toolkit for source code parsing, database creation, and database query  commercial grade front end parsers (EDG for C99/C++, Mutek for F90)  Intel/KAI C++ headers for std. C++ library distributed with PDT  portable IL analyzer, database format, and access API  open software approach for tool development  Target and integrate multiple source languages  Used in CCA for automated generation of SIDL  Used in TAU to build automated performance instrumentation tools (tau_instrumentor)  Used in CHASM, XMLGEN, Component method signature extraction,…

21. University of Vienna 21 Jan. 22, 2003 TAU’s Runtime Monitor Application Performance Steering Performance Visualizer Performance Analyzer Performance Data Reader TAU Performance System Performance Data Integrator SCIRun // performance data streams // performance data output file system sample sequencing reader synchronization

22. University of Vienna 22 Jan. 22, 2003 2D Field Performance Visualization in SCIRun SCIRun program

23. University of Vienna 23 Jan. 22, 2003 TAU’s Runtime Monitor TAU uses SCIRun [U. Utah] for visualization of performance data (online/offline)

24. University of Vienna 24 Jan. 22, 2003 Performance Database Framework... Raw performance data PerfDML data description Performance analysis programs PerfDML translators Performance analysis and query toolkit ORDB PostgreSQL XML profile data representation Multiple experiment performance database

25. University of Vienna 25 Jan. 22, 2003 Performance Tracking and Reporting  Integrated performance measurement allows performance analysis throughout development lifetime  Applied performance engineering in software design and development (software engineering) process  Create “performance portfolio” from regular performance experimentation (couple with software testing)  Use performance knowledge in making key software design decision, prior to major development stages  Use performance benchmarking and regression testing to identify irregularities  Support automatic reporting of “performance bugs”  Enable cross-platform (cross-generation) evaluation

26. University of Vienna 26 Jan. 22, 2003 XPARE System Architecture Experiment Launch Mail server Performance Database Performance Reporter Comparison Tool Regression Analyzer Alerting Setup Web server

27. University of Vienna 27 Jan. 22, 2003 Outline  Overview of the TAU project  Performance analysis of distributed Java programs  Profiling toolkit, PDT, Database, Monitor, XPARE  Performance Engineered Component Software  CCA Performance Observation Component  CCAFFEINE (Classic C++)  SIDL  Applications:  Optimizer Component  Combustion Component  Concluding remarks

28. University of Vienna 28 Jan. 22, 2003 Motivation for Parallel Component Software  History of HPC reflects evolving complexity of parallel and distributed systems used for scientific computing  Application development environments leverage power of software abstraction in scientific problem solving  Natural tension between achieving high performance and software engineering for scientific computing  Common dogma: further software is away from the raw machine, the harder it is to achieve good performance  Strategies include layered infrastructure with rich middleware support implemented for high-performance  Compromise is to further distance application developer from broader range performance sources and problems

29. University of Vienna 29 Jan. 22, 2003 Scientific Component Software  Emerging use for scientific high-performance computing  Targets spectrum of HPC systems and applications  Realistic programming and computing model for the Grid  Pose challenges for performance engineering  Software and system diversity  Flexibility in construction and connection  Language interoperability  Platform interoperability  Architecture vs. implementation  New environment for performance problem solving?  Does HPC performance experience / technology apply?

30. University of Vienna 30 Jan. 22, 2003 Component Technology  What is a component?  Implementation provides functionality buts hides details  No direct access is possible  Interface provides access to component functionality  Access “ports” are well-defined and generated by tools  Matching connector links component interfaces  Constructed by framework and hidden from users

31. University of Vienna 31 Jan. 22, 2003 Component Technology Features  Interoperability across multiple languages  Language independent interfaces (C/C++, Fortran, Java,…)  Automatically generated bindings to working code  Interoperability across multiple platforms  Computer systems hardware independence  Operating systems independence  Transparent execution model  Serial, parallel, and distributed system  Incremental evolution of application software  Components promote software reuse  Components are “plug-and-play”

32. University of Vienna 32 Jan. 22, 2003 CCA Forum  Define specifications for high-performance scientific components and frameworks  Preservation of performance  Promote development of domain-specific “standard” interfaces  Goal: interoperability between components developed by different expert teams across different institutions  Quarterly meetings and open membership  Mailing list: cca-forum@cca-forum.orgcca-forum@cca-forum.org  Website: http://www.cca-forum.orghttp://www.cca-forum.org

33. University of Vienna 33 Jan. 22, 2003 DOE CCTTSS  DOE SciDAC ISIC  Scientific Discovery through Advanced Computing  Integrated Software Infrastructure Center  Subset of CCA Forum  Develop CCA technology from current prototype stage to full production environment  Increase understanding of how to use component architectures effectively in HPC environments  Participants:  Rob Armstrong, Lead, SNL

34. University of Vienna 34 Jan. 22, 2003 Common Component Architecture Specification Scientific IDL Proxy generator Component 1Component 2 CCA Services Any CCA compliant framework Builder Repository CCA ports Framework-specific part of CCA ports Abstract configuration API Repository API

35. University of Vienna 35 Jan. 22, 2003 CCA Concepts: Ports  Designing for interoperability and reuse requires “standard” interfaces  Ports define how components interact  Through well-defined interfaces (ports)  In OO languages, a port is a class or interface  In Fortran, a port is a set of subroutines or a module  Components may provide ports  Implement the class or subroutines of the port  Components may use ports  Call methods or subroutines in the port  Links denote a caller/callee relationship

36. University of Vienna 36 Jan. 22, 2003 CCA Concepts: Frameworks  Provides the means to “hold” components and compose them into applications  Allow exchange of ports among components without exposing implementation details  Provide a small set of standard services to components  Builder services allow programs to compose CCA apps  Frameworks may make themselves appear as components in order to connect to components in other frameworks  Specific frameworks support specific computing models

37. University of Vienna 37 Jan. 22, 2003 CCA Example  Numerically integrate a continuous function  Use two different techniques  Lines show port connections  Dashed lines are alternate port connections FunctionPort MidpointIntegrator IntegratorPort FunctionPort MonteCarloIntegrator IntegratorPort RandomGeneratorPort IntegratorPort Driver GoPort NonlinearFunction FunctionPort LinearFunction FunctionPort RandomGenerator RandomGeneratorPort PiFunction FunctionPort a b x x ab x n uniformily distributed over [a,b]

38. University of Vienna 38 Jan. 22, 2003  CCAFFEINE  SPMD/SCMD parallel, direct connect  Direct connection  CCAT / XCAT  Distributed network  Grid Web services  SCIRun  Parallel, multithreaded, direct connect  Decaf  Language interoperability via Babel  Legion (under development) CCA Framework Prototypes

39. University of Vienna 39 Jan. 22, 2003 Performance-Engineered Component Software  Intra- and Inter-component performance engineering  Four general parts:  Performance observation  integrated measurement and analysis  Performance query and monitoring  runtime access to performance information  Performance control  mechanisms to alter performance observation  Performance knowledge  characterization and modeling  Consistent with component architecture / implementation

40. University of Vienna 40 Jan. 22, 2003  Extend the programming and execution environment to be performance observable and performance aware Main Idea: Extend Component Design performance observation ports performance knowledge ports … Performance Knowledge Component Core … variants Performance Observation  empirical  analytical … … Component Performance Repository repository service ports component ports  measurement  analysis

41. University of Vienna 41 Jan. 22, 2003  Performance measurement integration in component form  Functional extension of original component design ( )  Include new component methods and ports ( ) for other components to access measured performance data  Allow original component to access performance data  Encapsulate as tightly-couple and co-resident performance observation object  POC “provides” port allow use of optimized interfaces ( ) to access ``internal'' performance observations performance observation ports … Component Core … variants Performance Observation … component ports  measurement  analysis Performance Observation and Component

42. University of Vienna 42 Jan. 22, 2003 Performance Knowledge  Describe and store “known” component performance  Benchmark characterizations in performance database  Empirical or analytical performance models  Saved information about component performance  Use for performance-guided selection and deployment  Use for runtime adaptation  Representation must be in common forms with standard means for accessing the performance information  Compatible with component architecture

43. University of Vienna 43 Jan. 22, 2003  Performance knowledge storage  Implement in component architecture framework  Similar to CCA component repository  Access by component infrastructure  View performance knowledge as component (PKC)  PKC ports give access to performance knowledge  to other components, back to original component  Static/dynamic component control and composition  Component composition performance knowledge Component Performance Repository performance knowledge ports Performance Knowledge  empirical  analytical … Component Performance Repository repository service ports

44. University of Vienna 44 Jan. 22, 2003 Performance Engineering Support in CCA  Define a standard observation component interface for:  Performance measurement  Performance data query  Performance control (enable/disable)  Implement performance interfaces for use in CCA  TAU performance system  CCA component frameworks (CCAFFEINE, SIDL/Babel)  Demonstrations  Optimizing component  picks from a set of equivalent CCA port implementations  Flame reaction-diffusion application

45. University of Vienna 45 Jan. 22, 2003 CCA Performance Observation Component  Design measurement port and measurement interfaces  Timer  start/stop  set name/type/group  Control  enable/disable groups  Query  get timer names  metrics, counters, dump to disk  Event  user-defined events

46. University of Vienna 46 Jan. 22, 2003 CCA C++ (CCAFFEINE) Performance Interface namespace performance { namespace ccaports { class Measurement: public virtual classic::gov::cca::Port { public: virtual ~ Measurement (){} /* Create a Timer interface */ virtual performance::Timer* createTimer(void) = 0; virtual performance::Timer* createTimer(string name) = 0; virtual performance::Timer* createTimer(string name, string type) = 0; virtual performance::Timer* createTimer(string name, string type, string group) = 0; /* Create a Query interface */ virtual performance::Query* createQuery(void) = 0; /* Create a user-defined Event interface */ virtual performance::Event* createEvent(void) = 0; virtual performance::Event* createEvent(string name) = 0; /* Create a Control interface for selectively enabling and disabling * the instrumentation based on groups */ virtual performance::Control* createControl(void) = 0; }; } Measurement port Measurement interfaces

47. University of Vienna 47 Jan. 22, 2003 CCA Timer Interface Declaration namespace performance { class Timer { public: virtual ~Timer() {} /* Implement methods in a derived class to provide functionality */ /* Start and stop the Timer */ virtual void start(void) = 0; virtual void stop(void) = 0; /* Set name and type for Timer */ virtual void setName(string name) = 0; virtual string getName(void) = 0; virtual void setType(string name) = 0; virtual string getType(void) = 0; /* Set the group name and group type associated with the Timer */ virtual void setGroupName(string name) = 0; virtual string getGroupName(void) = 0; virtual void setGroupId(unsigned long group ) = 0; virtual unsigned long getGroupId(void) = 0; }; } Timer interface methods

48. University of Vienna 48 Jan. 22, 2003 Use of Observation Component in CCA Example #include "ports/Measurement_CCA.h"... double MonteCarloIntegrator::integrate(double lowBound, double upBound, int count) { classic::gov::cca::Port * port; double sum = 0.0; // Get Measurement port port = frameworkServices->getPort ("MeasurementPort"); if (port) measurement_m = dynamic_cast (port); if (measurement_m == 0){ cerr << "Connected to something other than a Measurement port"; return -1; } static performance::Timer* t = measurement_m->createTimer( string("IntegrateTimer")); t->start(); for (int i = 0; i getRandomNumber (); sum = sum + function_m->evaluate (x); } t->stop(); }

49. University of Vienna 49 Jan. 22, 2003 Using TAU Component in CCAFEINE repository get TauTimer repository get Driver repository get MidpointIntegrator repository get MonteCarloIntegrator repository get RandomGenerator repository get LinearFunction repository get NonlinearFunction repository get PiFunction create LinearFunction lin_func create NonlinearFunction nonlin_func create PiFunction pi_func create MonteCarloIntegrator mc_integrator create RandomGenerator rand create TauTimer tau connect mc_integrator RandomGeneratorPort rand RandomGeneratorPort connect mc_integrator FunctionPort nonlin_func FunctionPort connect mc_integrator TimerPort tau TimerPort create Driver driver connect driver IntegratorPort mc_integrator IntegratorPort go driver Go quit

50. University of Vienna 50 Jan. 22, 2003 Measurement Port Implementation  Use of Measurement port (i.e., instrumentation)  independent of choice of measurement tool  independent of choice of measurement type  TAU performance observability component  Implements the Measurement port  Implements Timer, Control, Query, Control  Port can be registered with the CCAFEINE framework  Components instrument to generic Measurement port  Runtime selection of TAU component during execution  TauMeasurement_CCA port implementation uses a specific TAU library for choice of measurement type

51. University of Vienna 51 Jan. 22, 2003 What’s Going On Here? TAU API runtime TAU performance data TAU API application component performance component other API … Alternative implementations of performance component Two instrumentation paths using TAU API Two query and control paths using TAU API application component

52. University of Vienna 52 Jan. 22, 2003 SIDL Interface for Performance Component version performance 1.0; package performance { interface Timer { /* Start/stop the Timer */ void start(); void stop(); /* Set/get the Timer name */ void setName(in string name); string getName(); /* Set/get Timer type information (e.g., signature of the routine) */ void setType(in string name); string getType(); /* Set/get the group name associated with the Timer */ void setGroupName(in string name); string getGroupName(); /* Set/get the group id associated with the Timer */ void setGroupId(in long group); long getGroupId(); } …

53. University of Vienna 53 Jan. 22, 2003 Simple Runtime Performance Optimization  Components are “plug-and-play”  One can choose from a set of equivalent port implementations based on performance measurements  An outside agent can monitor and select an optimal working set of components FunctionPort MidpointIntegrator IntegratorPort FunctionPort MonteCarloIntegrator IntegratorPort RandomGeneratorPort IntegratorPort Driver GoPort NonlinearFunction FunctionPort LinearFunction FunctionPort RandomGenerator RandomGeneratorPort PiFunction FunctionPort

54. University of Vienna 54 Jan. 22, 2003 Component Optimizing Performance Results

55. University of Vienna 55 Jan. 22, 2003 Computational Facility for Reacting Flow Science  Sandia National Laboratory  DOE SciDAC project (http://cfrfs.ca.sandia.gov)http://cfrfs.ca.sandia.gov  Jaideep Ray  Component-based simulation and analysis  Sandia’s CCAFFEINE framework  Toolkit components for assembling flame simulation  integrator, spatial discretizations, chemical/transport models  structured adaptive mesh, load-balancers, error-estimators  in-core, off-machine, data transfers for post-processing  Components are C++ and wrapped F77 and C code  Kernel for 3D, adaptive mesh low Mach flame simulation

56. University of Vienna 56 Jan. 22, 2003 Flame Reaction-Diffusion Demonstration CCAFFEINE

57. University of Vienna 57 Jan. 22, 2003 Meeting CCA Performance Engineering Goals?  Language interoperability?  SIDL and Babel give access to all supported languages  TAU supports multi-language instrumentation  Component interface instrumentation automated with PDT  Platform interoperability?  Implement observability component across platforms  TAU runs wherever CCA runs  Execution model transparent?  TAU measurement support for multiple execution models  Reuse with any CCA-compliant framework?  Demonstrated with SIDL/Babel, CCAFEINE, SCIRun

58. University of Vienna 58 Jan. 22, 2003 Importance to Grid Computing and Performance  Component software is a natural model for developing applications for the Grid  ICENI (Imperial College), CCAT / XCAT (U. Indiana)  Our work leverages abstraction power of CCA as well as the infrastructure of CCA frameworks  Similarly leverage Grid infrastructure and services  Mostly riding back of CCA framework development  Application-level performance view coupled with Grid resource assessment and monitoring  More responsive to performance dynamics  Beginning work with NWS forecaster in applications

59. University of Vienna 59 Jan. 22, 2003 Meeting CCA Performance Engineering Goals?  Component performance knowledge?  Representation and performance repository work to do  Utilize effectively for deployment and steering  Build repository with TAU performance database  Performance of component compositions?  Component-to-component performance  Per connection instrumentation and measurement  Utilize performance mapping support  Ensemble-wide performance monitoring  connect performance “producers” to “consumers”  component-style implementation

60. University of Vienna 60 Jan. 22, 2003 Concluding Remarks  Parallel component systems pose challenging performance analysis problems that require robust methodologies and tools  New performance problems will arise  Instrumentation and measurement  Data analysis and presentation  Diagnosis and tuning  Performance modeling  Performance engineered components  Performance knowledge, observation, query and control

61. Support Acknowledgement  TAU and PDT support:  Department of Energy (DOE)  DOE 2000 ACTS contract  DOE MICS contract  DOE ASCI Level 3 (LANL, LLNL)  U. of Utah DOE ASCI Level 1 subcontract  DARPA  NSF National Young Investigator (NYI) award