1. Methodology easy but important 1

2. ToC 1.What is performance evaluation about ? 2.Metrics, Load and Goals 3.Hidden Factors 4.The Scientific Method 5.Patterns 2

3. What is Performance Evaluation ?

4. Load You need to define the load under which your system operates Make the difference between Intensity of the load (e.g. nb jobs per second) Nature of the load Statistical details that may matter: e.g. job sizes are heavy tailed or not Benchmarks are artificial load generators; we will play with one of them

5. Metric Define a metric; examples Response time Power consumption Throughput Define operational conditions under which metric is measured (« Viewpoint », see Chapter 11)

6. Compare Windows vs Linux 6

7. 7 Syscall Benchmark

8. 8 Memory Access Time

9. 9 Ghostscript

10. Metrics are often Multidimensional 10

11. A and D are non dominated 11

12. Know your goals A1 and A3 are comparisons, A2 is an absolute statement E2 is an engineering rule

13. 3. Hidden Factors Factor: an element that may impact the performance (desired factors): intensity of load, number of servers (nuisance factors): time of the day, presence of denial of service attack 13

14. TCP Throughput Increases with Mobility

15. TCP Throughput Decreases with Mobility

16. Why were we fooled ? Hidden factor had a more important role than the factor we were interested in We interpreted correlation as causality Need to be aware of all factors and incorporate in the analysis Or randomize experiment to reduce impact of hidden factors

17. Simpson’s Paradox A well known phenomenon -- Special case of Hidden Factor paradox when metric is success rate and factors are discrete

18. Berkeley Sex Case 1973 (source: wikipedia) 18

19. Take Home Message Pitfall number 1 is the presence of hidden factor Any study is susceptible to it Easy for opponents to find

20. 4. Be Scientific Joe measures performance of his Wireless Shop: what would you conclude ?

21. Scientific Method Joe buys 2 more Access Points improvement ? 21 BeforeAfter

22. Scientific Method A conclusion can only proven to be wrong Do not draw conclusions unless the experiment was designed to test the statement Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement Joe should: design an experiment to validate: H1: “the wireless network is the bottleneck” for example: measure the number of collisions / packet loss result: collision · 1%; conclusion: H1 is not valid hypothesis H2: the server is saturated experiment: measure memory utilization : result ¼ 100% 22

23. Performance After Doubling Server Memory 23

24. Example from Nitin Vaidya, Mobicom 2000 Tutorial, slides 298-299 24

25. 25

26. 26

27. 27

28. Use of Scientific Method Recognize a fact Pose a hypothesis Verify the hypothesis on simulations / measurements designed to test it TCP throughput may increase with mobility (1) Duration of link failure period is impacted by speed (2) It has a negative impact on TCP throughput Do more simulations measure distrib of link failure period Verify (1) and (2) How ? 28

29. Is ATM-ABR better than ATM-UBR ? 29

30. Take Home Message You should not conclude from an experiment without trying to invalidate the conclusion (Popper, 1934): you should alternate between the roles of Proponent Adversary

31. 5. Patterns These are common traits found in different situations Knowing some of them may save a lot of time

33. Bottlenecks may be your enemy Bottlenecks are like non invited people at a party – they may impose their agenda Previous example: what we are measuring is the bottleneck, not the intended factor 33

34. Bottlenecks are Your Friends

35. Simplify your life, analyze bottlenecks ! In many cases, you may ignore the rest 35

36. Behind a Bottleneck May Hide Another Bottleneck

37. 37 i ’ i ” i

38. i ’ i ” i 38

39. 39

40. Congestion Collapse Definition: Offered load increases, work done decreases Frequent in complex systems May be due to cost per job increases with load Impatience Rejection of jobs before completion Designer must do something to avoid congestion collapse Eg. Admission control in Apache servers Eg. TCP congestion control Analyst must look for congestion collapse 40

41. Sources use TCP (= fair scheduling). Increase capacity of link 5 to 100 kb/s; what happens ? 41

42. Competition Side Effect System balances resources according to some scheduling Apparent paradox: put more resources, some get less 42

43. No TCP, users send as much can Increase capacity of link 2 from 10 to 1000 kb/s 43 10 kb/s

44. Competition Side Effect Apparent paradox: put more resources, all get less 44

45. Museum Audio Guide 45 Low speed USB connections at docking station High speed

46. Latent Congestion Collapse System is susceptible to congestion collapse Low speed access prevents congestion collapse Adding resources reveals congestion collapse 46

47. Take Home Message Watch for patterns, they are very frequent Bottlenecks Congestion collapse Competition side effects Latent Congestion collapse 47

48. Now it’s your turn… 48