1 PERFORMANCE EVALUATION H Often one needs to design and conduct an experiment in order to: – demonstrate that a new technique or concept is feasible –demonstrate that a new method is better than an existing method –understand the impact of various factors and parameters on the overall system performance

2 PERFORMANCE EVALUATION H There is a whole field of computer science called computer systems performance evaluation that does exactly this H One of the best books is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 (listed in bibliography) H Much of what is outlined in this presentation is described in more detail in [Jain 1991]

3 PERF EVAL 101: THE BASICS H There are three main methods used in the design of performance studies H Experimental approaches –measurement and use of a real system H Analytic approaches –the use of mathematics, queueing theory, Petri Nets, abstract models, etc H Simulation approaches –design and use of computer simulations and simplified models to assess performance

4 EXPERIMENTAL DESIGN AND METHODOLOGY H The design of a performance study requires great care in experimental design and methodology H Need to identify –experimental factors to be tested –levels (settings) for these factors –performance metrics to be used –experimental design to be used

5 FACTORS H Factors are the main “components” or “things” that are to be varied in an experiment, because their impact on performance wants to be understood H Examples: switch size, network load, number of buffers at output ports H Need to choose factors properly, since the number of factors affects size of study

6 LEVELS H Levels are the precise settings of the factors that are to be used in an experiment H Examples: switch size N = 2, 4, 8, 16 H Example: buffer size B = 100, 200, 400, 800 H Need to choose levels realistically H Need to cover reasonable portion of the design space

7 PERFORMANCE METRICS H Performance metrics specify what you want to measure in your performance study H Examples: cell loss, cell delay H Must choose your metrics properly and instrument your experiment accordingly

8 EXPERIMENTAL DESIGN H Experimental design refers to the organizational structure of your experiment H Need to methodically go through factors and levels to get the full range of experimental results desired H There are several “classical” approaches to experimental design

9 EXAMPLES H One factor at a time –vary only one factor through its levels to see what the impact is on performance H Two factors at a time –vary two factors to see not only their individual effects, but also their interaction effects, if any H Full factorial –try every possible combination of factors and levels to see full range of performance results

10 OTHER ISSUES H Simulation run length –choosing a long enough run time to get statistically meaningful results (equilibrium) H Simulation start-up effects and end effects –deciding how much to “chop off” at the start and end of simulations to get proper results H Replications –ensure repeatability of results, and gain greater statistical confidence in the results given H Presentation of results

11 SUMMARY H Great care must be taken in experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood H Computer systems performance evaluation defines standard methods for designing and conducting performance studies H Please follow these guidelines (where applicable) when doing assignments and course projects