1. CPSC 531: Experiment Design1 CPSC 531: Experiment Design and Performance Evaluation Instructor: Anirban Mahanti Office: ICT 745 Email: mahanti@cpsc.ucalgary.ca Class Location: TRB 101 Lectures: TR 15:30 – 16:45 hours Class web page: http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 Slides courtesy Professor Carey Williamson (minor modifications by Anirban Mahanti).

2. CPSC 531: Experiment Design2 PERFORMANCE EVALUATION r Often one needs to design and conduct an experiment in order to: m demonstrate that a new technique or concept is feasible m demonstrate that a new method is better than an existing method m understand the impact of various factors and parameters on the overall system performance

3. CPSC 531: Experiment Design3 PERFORMANCE EVALUATION r There is a whole field of computer science called computer systems performance evaluation that does exactly this r One of the best books is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 (listed in bibliography) r Much of what is outlined in this presentation is described in more detail in [Jain 1991]

4. CPSC 531: Experiment Design4 PERF EVAL 101: THE BASICS r There are three main methods used in the design of performance studies r Experimental approaches m measurement and use of a real system r Analytic approaches m the use of mathematics, queueing theory, Petri Nets, abstract models, etc r Simulation approaches m design and use of computer simulations and simplified models to assess performance

5. CPSC 531: Experiment Design5 EXPERIMENTAL DESIGN AND METHODOLOGY r The design of a performance study requires great care in experimental design and methodology r Need to identify m experimental factors to be tested m levels (settings) for these factors m performance metrics to be used m experimental design to be used

6. CPSC 531: Experiment Design6 FACTORS r Factors are the main “components” or “things” that are to be varied in an experiment, because their impact on performance wants to be understood r Examples: switch size, network load, number of buffers at output ports r Need to choose factors properly, since the number of factors affects size of study

7. CPSC 531: Experiment Design7 LEVELS r Levels are the precise settings of the factors that are to be used in an experiment r Examples: switch size N = 2, 4, 8, 16 r Example: buffer size B = 100, 200, 400, 800 r Need to choose levels realistically r Need to cover reasonable portion of the design space

8. CPSC 531: Experiment Design8 PERFORMANCE METRICS r Performance metrics specify what you want to measure in your performance study r Examples: packet loss, packet delay r Must choose your metrics properly and instrument your experiment accordingly

9. CPSC 531: Experiment Design9 EXPERIMENTAL DESIGN r Experimental design refers to the organizational structure of your experiment r Need to methodically go through factors and levels to get the full range of experimental results desired r There are several “classical” approaches to experimental design

10. CPSC 531: Experiment Design10 Experiment Design - Classification r One factor at a time m vary only one factor through its levels to see what the impact is on performance r Two factors at a time m vary two factors to see not only their individual effects, but also their interaction effects, if any r Full factorial m try every possible combination of factors and levels to see full range of performance results

11. CPSC 531: Experiment Design11 Example: Web Proxy Caching Filtering Input stream Filtered stream (misses)

12. CPSC 531: Experiment Design12 Example: Factors and Levels r Cache size r Cache Replacement Policy m Recency-based LRU m Frequency-based LFU-Aging m Size-based GD-Size m RAND m FIFO r Workload Characteristics m One-timers, Zipf slope, tail index, correlation, temporal locality model

13. CPSC 531: Experiment Design13 Example: Performance Metrics r Document Hit Ratio (DHR) m What percentage of the documents requested by the clients are satisfied directly by the proxy, without having to obtain from the Web server? r Byte Hit Ratio (BHR) m What percentage of the bytes requested by the clients are satisfied directly by the proxy, without having to obtain from the Web server?

14. CPSC 531: Experiment Design14 Example: Performance Results Workload: Weak temporal localityWorkload: Strong temporal locality Document Hit Ratio

15. CPSC 531: Experiment Design15 OTHER ISSUES r Simulation run length m choosing a long enough run time to get statistically meaningful results (equilibrium) r Simulation start-up effects and end effects m deciding how much to “chop off” at the start and end of simulations to get proper results r Replications m ensure repeatability of results, and gain greater statistical confidence in the results given r Presentation of results

16. CPSC 531: Experiment Design16 Presentation of Results r Graphs and tables are the two most common ways of illustrating and/or summarizing data m graphs can show you the trends m tables provide the details r There are good ways and bad ways to do each of these r Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well

17. CPSC 531: Experiment Design17 Table Tips r Decide if a table is really needed; if so, should it be part of main report, or just an appendix? r Choose formatting software with which you are familiar; easy to import data, export tables r Table caption goes at the top r Clearly delineate rows and columns (lines) r Logically organize rows and columns r Report results to several significant digits r Be consistent in formatting wherever possible

18. CPSC 531: Experiment Design18 Graphing Tips r Choose a good software package, preferably one with which you are familiar, and one for which it is easy to import data, export graphs r Title at top; caption below (informative) r Labels on each axis, including units r Logical step sizes along axes (10’s, 100’s…) r Make sure choice of scale is clear for each axis (linear, log-linear, log-log) r Graph should start from origin (zero) unless there is a compelling reason not to do so

19. CPSC 531: Experiment Design19 Graphing Tips (cont’d) r Make judicious choice of type of plot m scatter plot, line graph, bar chart, histogram r Make judicious choice of line types m solid, dashed, dotted, lines and points, colours r If multiple lines on a plot, then use a key, which should be well-placed and informative r If graph is “well-behaved”, then organize the key to match the lines on the graph (try it!) r Be consistent from one graph to the next wherever possible (size, scale, key, colours)

20. CPSC 531: Experiment Design20 SUMMARY r Great care must be taken in experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood r Computer systems performance evaluation defines standard methods for designing and conducting performance studies r Please follow these guidelines (where applicable) when doing assignments and course projects