Evaluating Performance Information for Mapping Algorithms to Advanced Architectures Nayda G. Santiago, PhD, PE Electrical and Computer Engineering Department.

Slides:



Advertisements
Similar presentations
Educational Research: Causal-Comparative Studies
Advertisements

Design of Experiments Lecture I
Aaker, Kumar, Day Seventh Edition Instructor’s Presentation Slides
Analysis of Variance Outlines: Designing Engineering Experiments
Experimental Design, Response Surface Analysis, and Optimization
Analysis of variance (ANOVA)-the General Linear Model (GLM)
Chapter 10 Analysis of Variance (ANOVA) Part III: Additional Hypothesis Tests Renee R. Ha, Ph.D. James C. Ha, Ph.D Integrative Statistics for the Social.
Chapter 13 Introduction to Linear Regression and Correlation Analysis
Causal Comparative Research: Purpose
Lecture 9: One Way ANOVA Between Subjects
Statistics: The Science of Learning from Data Data Collection Data Analysis Interpretation Prediction  Take Action W.E. Deming “The value of statistics.
EEM332 Design of Experiments En. Mohd Nazri Mahmud
Outline Chapter 1 Hardware, Software, Programming, Web surfing, … Chapter Goals –Describe the layers of a computer system –Describe the concept.
On Fairness, Optimizing Replica Selection in Data Grids Husni Hamad E. AL-Mistarihi and Chan Huah Yong IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
Chapter 14 Introduction to Linear Regression and Correlation Analysis
T WO WAY ANOVA WITH REPLICATION  Also called a Factorial Experiment.  Replication means an independent repeat of each factor combination.  The purpose.
Microarray Gene Expression Data Analysis A.Venkatesh CBBL Functional Genomics Chapter: 07.
Section 2: Science as a Process
Fundamentals of Data Analysis Lecture 7 ANOVA. Program for today F Analysis of variance; F One factor design; F Many factors design; F Latin square scheme.
Unit 2: Engineering Design Process
1 Chapter 1: Introduction to Design of Experiments 1.1 Review of Basic Statistical Concepts (Optional) 1.2 Introduction to Experimental Design 1.3 Completely.
QNT 531 Advanced Problems in Statistics and Research Methods
S519: Evaluation of Information Systems Week 14: April 7, 2008.
INT 506/706: Total Quality Management Introduction to Design of Experiments.
T tests comparing two means t tests comparing two means.
Topics: Statistics & Experimental Design The Human Visual System Color Science Light Sources: Radiometry/Photometry Geometric Optics Tone-transfer Function.
The Research Enterprise in Psychology. The Scientific Method: Terminology Operational definitions are used to clarify precisely what is meant by each.
© 2002 Prentice-Hall, Inc.Chap 9-1 Statistics for Managers Using Microsoft Excel 3 rd Edition Chapter 9 Analysis of Variance.
Analyzing and Interpreting Quantitative Data
Chap 12-1 A Course In Business Statistics, 4th © 2006 Prentice-Hall, Inc. A Course In Business Statistics 4 th Edition Chapter 12 Introduction to Linear.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
HOW TO WRITE RESEARCH PROPOSAL BY DR. NIK MAHERAN NIK MUHAMMAD.
Experimental Design If a process is in statistical control but has poor capability it will often be necessary to reduce variability. Experimental design.
The Research Design. Experimental Design Definition A description of what a researcher would like to find out and how to find it out. Pre-requisites 1.Identification.
Data Reduction. 1.Overview 2.The Curse of Dimensionality 3.Data Sampling 4.Binning and Reduction of Cardinality.
1 Chapter 1: Introduction to Design of Experiments 1.1 Review of Basic Statistical Concepts (Optional) 1.2 Introduction to Experimental Design 1.3 Completely.
ANALYSIS OF VARIANCE (ANOVA) BCT 2053 CHAPTER 5. CONTENT 5.1 Introduction to ANOVA 5.2 One-Way ANOVA 5.3 Two-Way ANOVA.
Statistical Analysis of Inlining Heuristics in Jikes RVM Jing Yang Department of Computer Science, University of Virginia.
ITEC6310 Research Methods in Information Technology Instructor: Prof. Z. Yang Course Website: c6310.htm Office:
Introduction to Earth Science Section 2 Section 2: Science as a Process Preview Key Ideas Behavior of Natural Systems Scientific Methods Scientific Measurements.
1.1 Statistical Analysis. Learning Goals: Basic Statistics Data is best demonstrated visually in a graph form with clearly labeled axes and a concise.
Question paper 1997.
Experimentation in Computer Science (Part 2). Experimentation in Software Engineering --- Outline  Empirical Strategies  Measurement  Experiment Process.
Analyzing Expression Data: Clustering and Stats Chapter 16.
PCB 3043L - General Ecology Data Analysis.
Chapter 20 Classification and Estimation Classification – Feature selection Good feature have four characteristics: –Discrimination. Features.
Chapter 1 Introduction to Statistics. Section 1.1 Fundamental Statistical Concepts.
T tests comparing two means t tests comparing two means.
Statistics (cont.) Psych 231: Research Methods in Psychology.
Chapter 7: Hypothesis Testing. Learning Objectives Describe the process of hypothesis testing Correctly state hypotheses Distinguish between one-tailed.
Inferential Statistics Psych 231: Research Methods in Psychology.
NURS 306, Nursing Research Lisa Broughton, MSN, RN, CCRN RESEARCH STATISTICS.
BSHS 382 MASTER Leading through innovation/bshs382masterdotcom.
Descriptive Statistics Report Reliability test Validity test & Summated scale Dr. Peerayuth Charoensukmongkol, ICO NIDA Research Methods in Management.
STEP - 4 Research Design 1. The term “research design” can be defined as, The systematic study plan used to turn a research question or research questions.
Chapter 13 Simple Linear Regression
PCB 3043L - General Ecology Data Analysis.
Section 2: Science as a Process
Gerald Dyer, Jr., MPH October 20, 2016
Psych 231: Research Methods in Psychology
Psych 231: Research Methods in Psychology
Design Of Experiment Eng. Ibrahim Kuhail.
Psych 231: Research Methods in Psychology
Statistical Thinking and Applications
DESIGN OF EXPERIMENTS by R. C. Baker
14 Design of Experiments with Several Factors CHAPTER OUTLINE
STATISTICS INFORMED DECISIONS USING DATA
Presentation transcript:

Evaluating Performance Information for Mapping Algorithms to Advanced Architectures Nayda G. Santiago, PhD, PE Electrical and Computer Engineering Department University of Puerto Rico, Mayaguez Campus Sept 1, 2006

Outline Introduction Problems Methodology Objectives Previous Work Description of Methodology Case Study Results Conclusions Future Work

Introduction Problem solving on HPC facility Conceptualization Instantiation Mapping Parallel Implementation Goal Can we obtain metrics to characterize what is happening in the HPC system? Test a methodology for obtaining information from HPC system. Compare with current results.

Introduction Source Code CompilerLinker Executable File Mapping Process Libraries Running Program Instrumentation Measurement

Introduction Application Programmer Decisions Programming paradigm Language Compiler Libraries Advanced architecture Programming style Algorithms

Problems Different factors affect computer performance of an implementation. Information of high-level effects is lost in the mapping process Out of order execution Compiler optimizations Complex interactions of parallel code and systems Current performance analysis tools not appealing

Current Tuning Methodology High-level Code Computer System Instrumentation Tools Performance Data Analysis and Evaluation Tools Programmer LibrariesAlgorithms Programming Paradigm Programming Style Languages System Configuration Use Evaluation ExperienceKnowledge On Tools In-depth Knowledge On Computer System Understand Relations Between Performance Data and Code Burden on Programmer

New Tuning Methodology High-level Code Computer System Instrumentation Tools Performance Data Programmer Statistical Data Analysis Information Knowledge-Based System Suggestions Problem Solving Environment Experimentation Alternatives Modify

Proposed Tuning Methodology High-level Code Computer System Instrumentation Tools Performance Data Programmer Statistical Data Analysis Information Knowledge-Based System Suggestions Experimentation Alternatives Modify My work

Integrative Performance Analysis Abstraction – low-level information is hidden Problem Translation System Levels Metrics Machine OS Node Network Mapping back to user’s view? Measurement – low level information is collected User’s View Tools High-level Language Domain Factors

Objectives Obtain information on the relation between low-level performance information and factors that affect performance. Lessen the burden of the programmer to incorporate experience and knowledge into the tuning process. Identify the most important metrics describing system-software interactions. Identify how many metrics convey most of the information of the system.

Methodology Design of Experiment Data Analysis Data Collection Preliminary Problem Analysis

Methodology Design of Experiment Data Analysis Data Collection Preliminary Problem Analysis High-level Code Computer System Instrumentation Tools Performance Data Programmer Statistical Data Analysis Information Knowledge-Based System Suggestions Experimentation Alternatives Modify

Preliminary Problem Analysis Results Profiling is useful for preliminary analysis Contribution Screening is required to limit number of factors in experiment Feasibility Significance Due to the large number of factors affecting performance and the long running times, experimentation has not been commonly used for performance data evaluation. Screening will make it feasible. Preliminary Problem Analysis -Evaluation of alternatives -Screening experiment Factors for experimentation -Understanding of problem -Application -Performance goal -Potential factors affecting performance

Design of Experiment (DOE) Systematic planning of experiments Most information Minimize effect extraneous factors Causal relations Correlational relations Design of Experiment -Design Factors -Levels of each factor -Response variable -Choice of design -Order of treatments

Design of Experiment Three basic principles Replication Estimate experimental error Precision Randomization Independence between observations Average out effect extraneous factors Blocking Block – Set homogeneous experimental conditions

Design of Experiment Results The appropriate randomization scheme, number of replications, and treatment order for the experimental runs. Contributions Innovative use of DOE for establishing causal relations for application tuning The appropriate design of experiment should be selected according to the performance analysis problem Significance The use of DOE and ANOVA will determine the cause of the performance differences in the results

Data Collection Instrumentation Dependent on the system Observable computing system Metrics can be observed in the system Data Collection -Executable File -System -Instrumentation Tool -Raw data -Sampling -Profiles

Data Collection Instrumentation Software Hardware Instrumentation tool setup Experimental runs and data collection

Data Collection Results Measurements of the metrics observed from the system Particular to this case study Between 36 and 52 metrics Data Collection -Tool Configuration -Order of runs -Crontab file -System Raw data (metrics)

Data Analysis Statistical Analysis Correlation matrix Multidimensional methods Dimensionality estimation Subset Selection Entropy cost function ANOVA Post hoc comparisons Data Analysis Raw Data Information

Data Analysis Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Data Analysis: Data Conversion Raw data Sampling Profiling Performance Data Matrix Random process Average Random variable Convert Format Raw Data Performance Data Matrix

Data Analysis: Data Conversion Performance data matrix m a (k,p), where: a: abs or avg k: experimental run p: metric identification number Multidimensional M = m a [0,0]m a [0,1] m a [1,0] m a [K-1,0] m a [1,1] m a [0,P-1] m a [1,P-1] m a [K-1,P-1] m a [K-1,1] … … … … … … …

Data Analysis: Data Conversion Performance data matrix example M = ExecTime[0]Pgfaults/s[0] ExecTime[1] ExecTime[K-1] Pgfaults/s[1] IdleTime[0] IdleTime[1] IdleTime[K-1]Pgfaults/s[K-1] … … … … …… … Metric 0Metric 1Metric P-1 Run 0 Run 1 Run K-1

Data Analysis: Correlation Study Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Data Analysis: Correlation Study Correlation Measure of linear relation among variables No causal Correlation Matrix Performance Data Matrix Correlations

Data Analysis: Correlation Study Example

Data Analysis: Correlation Study Correlation formula Which metrics were most correlated with execution time Results of correlation analysis Collinearity Software instrumentation Where S x and S y are the sample estimate of the standard deviation

Data Analysis: Normalization Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Normalization Log normalization Min-max normalization Dimension normalization Data Analysis: Normalization Scales of metrics vary widely Normalize Performance Data Matrix Normalized Performance Data Matrix n a [k,p]=log(m a [k,p]) n a [k,p] = m a [k,p]-min(m p a [k]) max(m p a [k])-min(m p a [k]) n a [k,p]= m a [k,p] EuclNorm(m p a [k])

Data Analysis: Normalization Normalization Evaluation Artificially assign classes to data set Long execution time Short execution time Used visual separability criteria Principal Component Analysis (PCA) Project data along principal components Visualized separation of data

Data Analysis: Normalization Not Normalized

Data Analysis: Normalization Not Normalized

Data Analysis: Normalization Min-max normalization

Data Analysis: Normalization Normalizing to range (0,1)

Data Analysis: Normalization Normalizing with Euclidean Norm

Data Analysis: Normalization Normalizing with Euclidean Norm

Data Analysis: Normalization Results Appropriate normalization scheme Euclidean Normalization Contribution Usage of normalization schemes for performance data Significance Due to the effect of differences in scale, some statistical methods may be biased. By normalizing, results obtained will be due to the true nature of the problem and not caused by scale variations.

Data Analysis: Dimension Estimation Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Data Analysis: Dimension Estimation Dimensionality estimation How many metrics will explain the system’s behavior? Scree test Plot of eigenvalues of correlation matrix Cumulative Percentage of Total Variation Keep components explaining variance of data Kaiser-Guttman Eigenvalues of correlation matrix greater than one. Dimension P metrics K metrics K << P

Data Analysis: Dimension Estimation Example

Data Analysis: Dimension Estimation Results Dimension reduction to approximately 18% of the size All three methods have similar results Contribution Estimation of performance data sets dimension Significance Provides the minimum set of metrics that contain the most amount of information needed to evaluate the system

Data Analysis: Metric Subset Selection Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Data Analysis: Metric Subset Selection Subset Selection Sequential Forward Search Entropy Cost Function Subset Selection P metrics K metrics K << P where is the similarity value of two instances

Data Analysis: Metric Subset Selection Results Establishment of most important metrics For case study For experiment 1: Paging Activity For experiment 2: Memory faults For experiment 3: Buffer activity For experiment 4: Mix of metrics

Data Analysis: Metric Subset Selection Contributions The usage of: Feature subset selection to identify the most important metrics Entropy as a cost function for this purpose Significance The system is viewed as a source of information. If we can select metrics based on the amount of information they provide, we can narrow down the search for sources of performance problems.

Data Analysis: ANOVA Convert Format Normalize Correlation Matrix Dimension Subset Selection Anova Raw Data Post Hoc Comparisons Information Performance Data Matrix

Data Analysis: ANOVA Analysis of Variance (ANOVA) Cause of variations Null hypothesis Post Hoc Comparisons After null hypothesis is rejected Anova Post Hoc Comparisons Raw Data Factors Which level? How? Significant Differences? If factors Cause Variations

Data Analysis: ANOVA Results Set of factors affecting metric values and the values Contribution Use of ANOVA for analysis of performance metrics Significance ANOVA allows to identify whether the variations of the measurements are due to the random nature of the data or the factors. Incorrect conclusions may be reached if personal judgment is used.

Publications N. G. Santiago, D. T. Rover, and D. Rodriguez, "A Statistical Approach for the Analysis of the Relation Between Low- Level Performance Information, the Code, and the Environment", Information: An International Interdisciplinary Journal, Vol. 9, No 3, May 2006, pp N. G. Santiago, D. T. Rover, D. Rodriguez, “Subset Selection of Performance Metrics Describing System-Software Interactions”, SC2002, Supercomputing: High Performance Networking and Computing 2002, Baltimore MD, November , Santiago, N.G.; Rover, D.T.; Rodriguez, D., “A statistical approach for the analysis of the relation between low-level performance information, the code, and the environment”, The 4th Workshop on High Performance Scientific and Engineering Computing with Applications, HPSECA-02, Proceedings of the International Conference on Parallel Processing Workshops, August 18-21, 2002, Vancouver, British Columbia, Canada, Page(s):

Future Work Develop a means of providing feedback to the scientific programmer Design a knowledge-based system, PR system? Assign classes to performance outcomes and use a classifier Compare different entropy estimators for performance data evaluation. Evaluate other subset selection schemes Compare software versus hardware metrics Compare different architectures and programming paradigms

Questions?