1. TAU Performance DataBase Framework (PerfDBF)
Allen D. Malony
Department of Computer and Information Science
Computational Science Institute
University of Oregon

2. Outline Motivation for performance databases TAU performance system
TAU Performanc DataBase Framework
Architecture
XML profile data representation
Example
Performance engineering in software engineering
X-PARE (eXPeriment Alerting and Reporting)
Concluding remarks
The TAU Performance DataBase Framework

3. Why Performance Databases?
Focus on empirical performance optimization process
Necessary for multi-results performance analysis
Multiple experiments (codes, versions, platforms, …)
Historical performance comparison
Integral component of performance analysis framework
Improved performance analysis architecture design
More flexible and open tool interfaces
Supports extensibility and foreign tool interaction
Performance analysis collaboration
Performance tool sharing
Performance data sharing and knowledge base
The TAU Performance DataBase Framework

4. Empirical-Based Performance Optimization
Process
Experiment Schemas
Experiment
Trials
characterization
Performance Tuning
Performance
Diagnosis
Experimentation
Observation
hypotheses
properties ?
observability
requirements
The TAU Performance DataBase Framework

5. TAU Performance System Framework
Tuning and Analysis Utilities (aka Tools Are Us)
Performance system framework for scalable parallel and distributed high-performance computing
Targets a general complex system computation model
nodes / contexts / threads
Multi-level: system / software / parallelism
Measurement and analysis abstraction
Integrated toolkit for performance instrumentation, measurement, analysis, and visualization
Portable performance profiling/tracing facility
Open software approach
The TAU Performance DataBase Framework

6. TAU Performance System Architecture
Paraver
EPILOG
The TAU Performance DataBase Framework

7. TAU Performance Database Framework
Performance analysis programs
Performance
data description
Raw performance data
PerfDML
translators
Performance analysis
and query toolkit
. . .
ORDB
PostgreSQL
PerfDB
profile data only
XML representation
project / experiment / trial
The TAU Performance DataBase Framework

8. PerfDBF Components Performance Data Meta Language (PerfDML)
Common performance data representation
Performance meta-data description
PerfDML translators to common data representation
Performance DataBase (PerfDB)
Standard database technology (SQL)
Free, robust database software (PostgresSQL)
Commonly available APIs
Performance DataBase Toolkit (PerfDBT)
Commonly used modules for query and analysis
Facility analysis tool development
The TAU Performance DataBase Framework

9. Common and Extensible Profile Data Format
Goals
Capture data from profile tools in common representation
Implement representation in a standard format
Allow for extension of format for new profile data objects
Base on XML (obvious choice)
Leverage XML tools and APIs
XML parsers, Sun’s Java SDK, …
XML verification systems (DTD and schemas)
Target for profile data translation tools
eXtensibile Stylesheet Language Transformations (XSLT)
Which performance profile data are of interest?
Focus on TAU and consider other profiling tools
The TAU Performance DataBase Framework

10. Performance Profiling
Performance data about program entities and behaviors
Code regions: functions, loops, basic blocks
Actions or states
Statistics data
Execution time, number of calls, number of FLOPS ...
Characterization data
Parallel profiles
Captured per process and/or per thread
Program-level summaries
Profiling tools
prof/gprof, ssrun, uprofile/dpci, cprof/vprof, …
The TAU Performance DataBase Framework

11. TAU Parallel Performance Profiles
The TAU Performance DataBase Framework

12. PerfDBF Example NAS Parallel Benchmark LU
% configure -mpiinc=/usr/include -mpilib=/usr/lib64 -arch=sgi64 -fortran=sgi -SGITIMERS -useropt=-O2
NPB profiled
With TAU
Standard TAU
Output Data
TAU to XML
Converter
TAU XML
Format
Database Loader
SQL
Database
Analysis Tool
The TAU Performance DataBase Framework

13. Scalability Analysis Process
Scalability study on LU
% suite.def # of procs -> 1, 2, 4, and 8
% mpirun -np 1 lu.W1
% mpirun -np 2 lu.W2
% mpirun -np 4 lu.W4
% mpirun -np 8 lu.W8
populateDatabase.sh
run Java translator to translate profiles into XML
run Java XML reader to write XML profiles to database
Read times for routines and program from experiments
Calculate scalability metrics
The TAU Performance DataBase Framework

14. Raw TAU Profile Data Raw data output One processor: Four processors:
"applu ” GROUP="applu“
Four processors:
"applu ” GROUP="applu“
"applu " GROUP="applu“
"applu " GROUP="applu"
profile calls
group name
exclusive
time
inclusive
time
name
calls
subs
The TAU Performance DataBase Framework

15. XML Profile Representation
One processor
<instrumentedobj>
<funcname> 'applu '</funcname>
<funcID>8</funcID>
<inclperc>100.0</inclperc>
<inclutime> E8</inclutime>
<exclperc>0.0</exclperc>
<exclutime> </exclutime>
<call>1</call>
<subrs>15</subrs>
<inclutimePcall> E8</inclutimePcall>
</instrumentedobj>
The TAU Performance DataBase Framework

16. XML Representation Four processor mean <meanfunction>
<funcname>'applu '</funcname>
<funcID>12</funcID>
<inclperc>100.0</inclperc>
<inclutime> E7</inclutime>
<exclperc>0.0</exclperc>
<exclutime> </exclutime>
<call>1</call>
<subrs>14.25</subrs>
<inclutimePcall> E7</inclutimePcall>
</meanfunction>
The TAU Performance DataBase Framework

17. Contents of Performance Database
The TAU Performance DataBase Framework

18. Scalability Analysis Results
Scalability of LU performance experiments
Four trial runs
Funname | processors | meanspeedup ….
applu | 2 |
applu | 4 |
applu | 8 | …
exact | 2 |
exact | 4 |
exact | 8 |
The TAU Performance DataBase Framework

19. Current Status and Future
PerfDBF prototype
TAU profile to XML translator
XML to PerfDB populator
PostgresSQL database
Java-based PostgresSQL query module
Use as a layer to support performance analysis tools
Make accessing the Performance Database quicker
Continue development
XML parallel profile representation
Basic specification
Opportunity for APART to define a common format
The TAU Performance DataBase Framework

20. Performance Tracking and Reporting
Integrated performance measurement allows performance analysis throughout development lifetime
Applied performance engineering in software design and development (software engineering) process
Create “performance portfolio” from regular performance experimentation (couple with software testing)
Use performance knowledge in making key software design decision, prior to major development stages
Use performance benchmarking and regression testing to identify irregularities
Support automatic reporting of “performance bugs”
Enable cross-platform (cross-generation) evaluation
The TAU Performance DataBase Framework

21. XPARE - eXPeriment Alerting and REporting
Experiment launcher automates measurement / analysis
Configuration and compilation of performance tools
Instrumentation control for Uintah experiment type
Execution of multiple performance experiments
Performance data collection, analysis, and storage
Integrated in Uintah software testing harness
Reporting system conducts performance regression tests
Apply performance difference thresholds (alert ruleset)
Alerts users via if thresholds have been exceeded
Web alerting setup and full performance data reporting
Historical performance data analysis
The TAU Performance DataBase Framework

22. XPARE System Architecture
Experiment
Launch
Mail
server
Web
server
Performance
Database
Performance
Reporter
Alerting
Setup
Comparison
Tool
Regression
Analyzer
The TAU Performance DataBase Framework

23. Experiment Results Viewing Selection
The TAU Performance DataBase Framework

24. Web-Based Experiment Reporting
The TAU Performance DataBase Framework

25. Web-Based Experiment Reporting (continued)
The TAU Performance DataBase Framework

26. Alerting Setup
The TAU Performance DataBase Framework

27. Other Performance Database Projects
HPM Toolkit (DeRose; IBM)
PPerfDB (Karavanic; Portland State University)
HPCView (Mellor-Crummey, Fowler; Rice)
SCALEA (Fahringer, Truong; University of Vienna)
EXPERT (Mohr, Wolf; Research Center Juelich)
The TAU Performance DataBase Framework

28. Acknowledgements University of Oregon University of Utah
Li Li, Robert Bell, Sameer Shende
University of Utah
Alan Morris, Steve Parker, Dav St. Germain
Department of Energy (DOE), ASCI Academic Strategic Alliances Program (ASAP)
Computational Science Institute, ASCI/ASAP Level 3 projects with LLNL / LANL, University of Oregon
Center for the Simulation of Accidental Fires and Explosions (C-SAFE), ASCI/ASAP Level 1 center, University of Utah
The TAU Performance DataBase Framework