1. PAPI Tool Evaluation Bryan Golden 1/4/2004 HCS Research Laboratory University of Florida

2. 2 Basic Information Name: Performance Application Programming Interface (PAPI) Developer: University of Tennessee Knoxville Current version:  PAPI 3.0.7 Website: http://icl.cs.utk.edu/papi/http://icl.cs.utk.edu/papi/ Contact: Phil Mucci

3. 3 Evaluation (1) Available metrics (5 – standard set as well as comprehensive set)  HW counters Cost (5)  Free Documentation quality (5 – extensive user guide)  Clear document? Yes  Helpful document? Yes  Useful documents: PAPI 3 User’s Guide, Programmer’s Reference, Conversion Cookbook, and Software Specification Extendibility (5 – Many tools leverage the functionality of PAPI)  Not as well developed as MDL from Paradyne Filtering and Aggregation (N/A)

4. 4 PAPI Implementation Java Monitor GUI PAPI Low Level PAPI High Level Hardware Performance Counter Operating System Kernel Extension PAPI Machine Dependent Substrate Machine Specific Layer Portable Layer

5. 5 Evaluation (2) Hardware support (5) Heterogeneity support (4)  Yes – Code is highly portable, yet semantic differences may be troublesome HardwareOperating SystemRequirements POWER3, 604, 604eAIX4.3.xpmtoolkit from IBM alphaWorks IBM alphaWorks POWER4, POWER3, 604, 604e AIX 5.xinstall bos.pmapi Cray T3EUnicosNone Cray X1Unicos/mpNone AMD Athlon/Opteron, Intel thru Pentium III Linux 2.2,2.4,2.6Mikael Pettersson's PerfCtr kernel patch for Linux (included) Intel thru Pentium IIIWindows NT, 2000, XP adminstrator privilege for installation Intel Pentium IVLinux 2.2,2.4,2.6Mikael Pettersson's PerfCtr kernel patch for Linux (included) Intel Itanium I & IILinux 2.4,2.6Stephane Eranian's PerfMon kernel patch for Linux (included) UltraSparc I, II & IIISolaris 2.8,2.9None MIPS R10K, R12K, R14K IRIXNone AlphaTru64 UnixContact paul.drongowski@h p.com for required system software paul.drongowski@h p.com AlphaLinuxIProbe patch (included)

6. 6 Evaluation (3) Installation (3 – Requires kernel patch in some cases)  How to get software Download source from their website  How hard to install Simple with user guide Applying the patch can be a problem for some  Components needed Everything is included with the download  Estimate number of hours needed for installation Few hours with help from support Interoperability (5)  Other hardware counters: PCL  Other General tools: TAU, SCALEA, vprof, SvPablo, PeformanceBench, DynaProf

7. 7 Evaluation (4) Learning curve (3 –Very low level, but to be expected)  Very tedious but highly portable  The graphical interface, perfometer, is fairly well documented, however, it is not meant to be used as a tool (just to show off PAPI’s functionality) Manual overhead (2 – Function calls are almost like using printf, however, the naming conventions are intuitive)  Manual instrumentation Source code instrumentation with instrumentation API  No automatic instrumentation

8. 8 Evaluation (5) Measurement accuracy (5)  Evaluation of the measuring technique: The literature on the PAPI website provides validation of their techniques. However, due to the nature of HW counter data, this is almost trivial. Multiple analyses (1)  None Multiple executions (1 - PAPI is not designed for multiple executions)  No support Multiple views (1)  Views Profiling: perfometer has only one view, however, other tool which use PAPI may be better Intuitive? Yes, however, perfometer is not meant to be a tool Performance bottleneck identification (1)  No native support for multiple views Performance bottleneck identification (1)  None

9. 9 Evaluation (6) Profiling / tracing support (Various)  PAPI has no native tracing support, however many tools that use it do  Profiling? Tracing? Profiling only  Trace format? N/A  Trace strategy? N/A  Mechanism for turning on and off tracing N/A Searching (Various)  Dependent on the tool used to call the PAPI library routines Software support (N/A)  PAPI is a library

10. 10 Evaluation (7) Source code correlation (4)  Able to correlate performance data to source code: Can correlate back to the function name level but not to line numbers Response time (4)  No noticeable delays System stability (3)  Crash rate: Varies on different platforms, Windows XP crashes often, Linux is fairly stable Technical support (5)  Time to get a response from developer: Fast  Usefulness of messages the system provides: Good, messages are informative and thorough Comments  De facto standard for accessing HW counters  Works well with many other tools  Provides a great deal of information  Tool is fairly popular