Ratio Games and Designing Experiments Andy Wang CIS 5930-03 Computer Systems Performance Analysis.

Slides:



Advertisements
Similar presentations
Objectives 10.1 Simple linear regression
Advertisements

Exponential Functions Logarithmic Functions
Irwin/McGraw-Hill © Andrew F. Siegel, 1997 and l Chapter 12 l Multiple Regression: Predicting One Factor from Several Others.
Copyright © 2009 Pearson Education, Inc. Chapter 29 Multiple Regression.
IBM Labs in Haifa © 2005 IBM Corporation Adaptive Application of SAT Solving Techniques Ohad Shacham and Karen Yorav Presented by Sharon Barner.
Variance reduction techniques. 2 Introduction Simulation models should be coded such that they are efficient. Efficiency in terms of programming ensures.
Experimental Design, Response Surface Analysis, and Optimization
Fundamentals of Python: From First Programs Through Data Structures
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 13 Nonlinear and Multiple Regression.
1 Chapter 2 Simple Linear Regression Ray-Bing Chen Institute of Statistics National University of Kaohsiung.
Section 4.2 Fitting Curves and Surfaces by Least Squares.
MAE 552 Heuristic Optimization Instructor: John Eddy Lecture #19 3/8/02 Taguchi’s Orthogonal Arrays.
Introduction to Power Analysis  G. Quinn & M. Keough, 2003 Do not copy or distribute without permission of authors.
Statistics CSE 807.
10-1 ©2006 Raj Jain The Art of Data Presentation.
L Berkley Davis Copyright 2009 MER301: Engineering Reliability Lecture 14 1 MER301: Engineering Reliability LECTURE 14: Chapter 7: Design of Engineering.
Level ISA3: Information Representation
Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Introduction to Experiment Design Shiv Kalyanaraman Rensselaer Polytechnic Institute
Biol 500: basic statistics
k r Factorial Designs with Replications r replications of 2 k Experiments –2 k r observations. –Allows estimation of experimental errors Model:
Experimental Evaluation
Go to Table of ContentTable of Content Analysis of Variance: Randomized Blocks Farrokh Alemi Ph.D. Kashif Haqqi M.D.
LEARNING PROGRAMME Hypothesis testing Intermediate Training in Quantitative Analysis Bangkok November 2007.
Comparing Systems Using Sample Data Andy Wang CIS Computer Systems Performance Analysis.
EC339: Lecture 6 Chapter 5: Interpreting OLS Regression.
Data Collection & Processing Hand Grip Strength P textbook.
One-Factor Experiments Andy Wang CIS 5930 Computer Systems Performance Analysis.
CPE 619 2k-p Factorial Design
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Chapter 8: Estimating with Confidence Section 8.1 Confidence Intervals: The.
© 1998, Geoff Kuenning General 2 k Factorial Designs Used to explain the effects of k factors, each with two alternatives or levels 2 2 factorial designs.
Fundamentals of Data Analysis Lecture 10 Management of data sets and improving the precision of measurement pt. 2.
A Really Bad Graph. For Discussion Today Project Proposal 1.Statement of hypothesis 2.Workload decisions 3.Metrics to be used 4.Method.
Lecture 8 Page 1 CS 239, Spring 2007 Experiment Design CS 239 Experimental Methodologies for System Software Peter Reiher May 1, 2007.
Other Regression Models Andy Wang CIS Computer Systems Performance Analysis.
Introduction to Experimental Design
Test Loads Andy Wang CIS Computer Systems Performance Analysis.
BIOL 582 Lecture Set 11 Bivariate Data Correlation Regression.
© 1998, Geoff Kuenning Common Mistakes in Graphics Excess information Multiple scales Using symbols in place of text Poor scales Using lines incorrectly.
MSE-415: B. Hawrylo Chapter 13 – Robust Design What is robust design/process/product?: A robust product (process) is one that performs as intended even.
Intermediate Applied Statistics STAT 460 Lecture 17, 11/10/2004 Instructor: Aleksandra (Seša) Slavković TA: Wang Yu
© Copyright McGraw-Hill Correlation and Regression CHAPTER 10.
CPE 619 Experimental Design Aleksandar Milenković The LaCASA Laboratory Electrical and Computer Engineering Department The University of Alabama in Huntsville.
CROSS-VALIDATION AND MODEL SELECTION Many Slides are from: Dr. Thomas Jensen -Expedia.com and Prof. Olga Veksler - CS Learning and Computer Vision.
1 Overview of Experimental Design. 2 3 Examples of Experimental Designs.
Fitness Cases Are input-output pairs describing the output a program should produce given the particular input value. The fitness of an individual is usually.
Experiment Design Overview Number of factors 1 2 k levels 2:min/max n - cat num regression models2k2k repl interactions & errors 2 k-p weak interactions.
Fractional Factorial Designs Andy Wang CIS 5930 Computer Systems Performance Analysis.
Comparing Systems Using Sample Data Andy Wang CIS Computer Systems Performance Analysis.
Sampling Design and Analysis MTH 494 Lecture-21 Ossam Chohan Assistant Professor CIIT Abbottabad.
Multiple Linear Regression
Designs for Experiments with More Than One Factor When the experimenter is interested in the effect of multiple factors on a response a factorial design.
Chapter 7: Hypothesis Testing. Learning Objectives Describe the process of hypothesis testing Correctly state hypotheses Distinguish between one-tailed.
ChE 551 Lecture 04 Statistical Tests Of Rate Equations 1.
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Chapter 8: Estimating with Confidence Section 8.1 Confidence Intervals: The.
Engineering Statistics Design of Engineering Experiments.
BUS 308 Entire Course (Ash Course) For more course tutorials visit BUS 308 Week 1 Assignment Problems 1.2, 1.17, 3.3 & 3.22 BUS 308.
Test Loads Andy Wang CIS Computer Systems Performance Analysis.
Chapter 9 Introduction to the t Statistic
Stats Methods at IC Lecture 3: Regression.
Comparing Systems Using Sample Data
Two-Way Analysis of Variance Chapter 11.
Ratio Games and Designing Experiments
Two-Factor Full Factorial Designs
Correlation and Regression
CHAPTER 29: Multiple Regression*
Chapter 11 Analysis of Variance
Replicated Binary Designs
One-Factor Experiments
CS533 Modeling and Performance Evalua ion of Network and Computer Systems Experimental Design (Chapters 16-17)
Design of Experiments CHM 585 Chapter 15.
Presentation transcript:

Ratio Games and Designing Experiments Andy Wang CIS Computer Systems Performance Analysis

2 Ratio Games Choosing a base system Using ratio metrics Relative performance enhancement Ratio games with percentages Strategies for winning a ratio game Correct analysis of ratios

3 Choosing a Base System Run workloads on two systems Normalize performance to chosen system Take average of ratios Presto: you control what’s best

4 Example of Choosing a Base System (Carefully) selected Ficus results:

5 Why Does This Work? Expand the arithmetic:

6 Using Ratio Metrics Pick a metric that is itself a ratio –E.g., power = throughput  response time –Or cost/performance Handy because division is “hidden”

7 Relative Performance Enhancement Compare systems with incomparable bases Turn into ratios Example: compare Ficus 1 vs. 2 replicas with UFS vs. NFS (1 run on chosen day): “Proves” adding Ficus replica costs less than going from UFS to NFS

8 Ratio Games with Percentages Percentages are inherently ratios –But disguised –So great for ratio games Example: Passing tests A is worse, but looks better in total line!

9 More on Percentages Psychological impact –1000% sounds bigger than 10-fold (or 11- fold) –Great when both original and final performance are lousy E.g., salary went from $40 to $80 per week Small sample sizes generate big lies Base should be initial, not final value –E.g., price can’t drop 400%

10 Strategies for Winning a Ratio Game Can you win? How to win

11 Can You Win the Ratio Game? If one system is better by all measures, a ratio game won’t work –But recall percent-passes example –And selecting the base lets you change the magnitude of the difference If each system wins on some measures, ratio games might be possible (but no promises) –May have to try all bases

12 How to Win Your Ratio Game For LB metrics, use your system as the base For HB metrics, use the other as a base If possible, adjust lengths of benchmarks –Elongate when your system performs best –Short when your system is worst –This gives greater weight to your strengths

13 Correct Analysis of Ratios Previously covered in Lecture 2 Generally, harmonic or geometric means are appropriate –Or use only the raw data

14 Introduction To Experimental Design You know your metrics You know your factors You know your levels You’ve got your instrumentation and test loads Now what?

15 Goals in Experiment Design Obtain maximum information with minimum work –Typically meaning minimum number of experiments More experiments aren’t better if you’re the one who has to perform them Well-designed experiments are also easier to analyze

16 Experimental Replications System under study will be run with varying levels of different factors, potentially with differing workloads Run with particular set of levels and other inputs is a replication Often, need to do multiple replications with each set of levels and other inputs –Usually necessary for statistical validation

17 Interacting Factors Some factors have completely independent effects –Double the factor’s level, halve the response, regardless of other factors But effects of some factors depends on values of others –Called interacting factors Presence of interacting factors complicates experimental design

18 The Basic Problem in Designing Experiments You’ve chosen some number of factors –May or may not interact How to design experiment that captures full range of levels? –Want minimum amount of work Which combination or combinations of levels (of factors) do you measure?

19 Common Mistakes in Experimentation Ignoring experimental error Uncontrolled parameters Not isolating effects of different factors One-factor-at-a-time experimental designs Interactions ignored Designs require too many experiments

20 Types of Experimental Designs Simple designs Full factorial design Fractional factorial design

21 Simple Designs Vary one factor at a time For k factors with i th factor having n i levels, no. of experiments needed is: Assumes factors don’t interact –Even then, more effort than required Don’t use it, usually

22 Full Factorial Designs Test every possible combination of factors’ levels For k factors with i th factor having n i levels: Captures full information about interaction But a huge amount of work

23 Reducing the Work in Full Factorial Designs Reduce number of levels per factor –Generally good choice –Especially if you know which factors are most important Use more levels for those Reduce number of factors –But don’t drop important ones! Use fractional factorial designs

24 Fractional Factorial Designs Only measure some combination of levels of the factors Must design carefully to best capture any possible interactions Less work, but more chance of inaccuracy Especially useful if some factors are known to not interact

25 2 k Factorial Designs Used to determine effect of k factors –Each with two alternatives or levels Often used as preliminary to larger performance study –Each factor measured at its maximum and minimum level –Might offer insight on importance and interaction of various factors

26 Unidirectional Effects Effects that only increase as level of a factor increases –Or vice versa If system known to have unidirectional effects, 2 k factorial design at minimum and maximum levels is useful Shows whether factor has significant effect

Factorial Designs Two factors with two levels each Simplest kind of factorial experiment design Concepts developed here generalize Regression can easily be used

Factorial Design Example Consider parallel operating system Goal is fastest possible completion of a given program Quality usually expressed as speedup We’ll use runtime as metric (simpler but equivalent)

29 Factors and Levels for Parallel OS First factor: number of CPUs –Vary between 8 and 64 Second factor: use of dynamic load management –Migrates work between nodes as load changes Other factors possible, but ignore them for now

30 Defining Variables for 2 2 Factorial OS Example if 8 nodes if 64 nodes if no dynamic load mgmt if dynamic load mgmt

31 Sample Data for Parallel OS Single runs of one benchmark DLM NO DLM 8 Nodes64 Nodes

32 Regression Model for Example y = q 0 + q A x A + q B x B + q AB x A x B Note that model is nonlinear! 820 = q 0 – q A – q B + q AB 217 = q 0 + q A – q B – q AB 776 = q 0 – q A + q B – q AB 197 = q 0 + q A + q B + q AB

33 Solving the Equations 4 equations in 4 unknowns q 0 = q A = q B = -16 q AB = 6 So y = – 295.5x A – 16x B + 6x A x B

34 The Sign Table Method Write problem in tabular form IABAB y Total Total/4

35 Allocation of Variation for 2 2 Model Calculate the sample variance of y: Numerator is SST: total variation SST = 2 2 q A q B q AB 2 SST explains causes of variation in y

36 Terms in the SST 2 2 q A 2 is variation explained by effect of A: SSA 2 2 q B 2 is variation explained by effect of B: SSB 2 2 q AB 2 is variation explained by interaction between A and B: SSAB SST = SSA + SSB + SSAB

37 Variations in Our Example SST = SSA = SSB = 1024 SSAB = 144 Now easy to calculate fraction of total variation caused by each effect

38 Fractions of Variation in Our Example Fraction explained by A is 99.67% Fraction explained by B is 0.29% Fraction explained by interaction of A and B is 0.04% So almost all variation comes from number of nodes If you want to run faster, apply more nodes, don’t turn on dynamic load management

39 General 2 k Factorial Designs Used to explain effects of k factors, each with two alternatives or levels 2 2 factorial designs are a special case Same methods extend to more general case Many more interactions between pairs (and trios, etc.) of factors

40 Sample 2 3 Experiment Sign table A, B, C are binary count; interactions are products of columns:

White Slide