1. OS Fall ’ 02 Performance Evaluation Operating Systems Fall 2002

2. OS Fall ’ 02 Performance evaluation  There are several approaches for implementing the same OS functionality Different scheduling algorithms Different memory management schemes  Performance evaluation deals with the question how to compare wellness of different approaches Metrics, methods for evaluating metrics

3. OS Fall ’ 02 Performance Metrics  Is something wrong with the following statement: The complexity of my OS is O(n)?  This statement is inherently flawed The reason: OS is a reactive program  What is the performance metric for the sorting algorithms?

4. OS Fall ’ 02 Performance metrics  Response time  Throughput  Utilization  Other metrics: Mean Time Between Failures (MTBF) Supportable load

5. OS Fall ’ 02 Response time  The time interval between a user ’ s request and the system response Response time, reaction time, turnaround time, etc.  Small response time is good: For the user: waiting less For the system: free to do other things

6. OS Fall ’ 02 Throughput  Number of work units done per time unit Applications being run, files transferred, etc.  High throughput is good For the system: was able to serve many clients For the user: might imply worse service

7. OS Fall ’ 02 Utilization  Percentage of time the system is busy servicing clients Important for expensive shared system Less important (if at all)  for single user systems, for real time systems  Utilization and response time are interrelated Very high utilization may negatively affect response time

8. OS Fall ’ 02 Performance evaluation methods  Mathematical analysis Based on a rigorous mathematical model  Simulation Simulate the system operation (usually only small parts thereof)  Measurement Implement the system in full and measure its performance directly

9. OS Fall ’ 02 Analysis: Pros and Cons + Provides the best insight into the effects of different parameters and their interaction Is it better to configure the system with one fast disk or with two slow disks? + Can be done before the system is built and takes a short time - Rarely accurate Depends on host of simplifying assumptions

10. OS Fall ’ 02 Simulation: Pros and Cons + Flexibility: full control of Simulation model, parameters, Level of detail  Disk: average seek time vs. acceleration and stabilization of the head + Can be done before the system is built - Simulation of a full system is infeasible - Simulation of the system parts does not take everything into account

11. OS Fall ’ 02 Measurements: Pros and Cons + The most convincing - Effects of varying parameter values cannot (if at all) be easily isolated Often confused with random changes in the environment - High cost: Implement the system in full, buy hardware

12. OS Fall ’ 02 The bottom line  Simulation is the most widely used technique  Combination of techniques Never trust the results produced by the single method  Validate with another one  E.g., simulation + analysis, simulation + measurements, etc.

13. OS Fall ’ 02 Workload  Workload is the sequence of things to do Sequence of jobs submitted to the system  Arrival time, resources needed File system: Sequence of I/O operations  Number of bytes to access  Workload is the input of the reactive system The system performance depends on the workload

14. OS Fall ’ 02 Workload analysis  Workload modeling Use past measurements to create a model  E.g., fit them into a distribution Analysis, simulation, measurement  Recorded workload Use past workload directly to drive evaluation Simulation, measurement

15. OS Fall ’ 02 Statistical characterization  Every workload item is sampled at random from the distribution of some random variable Workload is characterized by a distribution E.g., take all possible job times and fit them to a distribution

16. OS Fall ’ 02 The Exponential Distribution  A lot of low values and a few high values  The distributions of salaries, lifetimes, and waiting times are often fit the exponential distribution  The distribution of Job runtimes Job inter-arrival times File sizes

17. OS Fall ’ 02 Exponential Probability Density Function (pdf)  If X has an exponential distribution with parameter, then its probability density function is given by: where

18. OS Fall ’ 02 The Exponential Distribution

19. OS Fall ’ 02 Mean and Variance  If X~Exponential( ), then its mean and variance are given by:

20. OS Fall ’ 02 Exponential Probabilities a x f(x) a x 00

21. OS Fall ’ 02 Cumulative Distribution Function  The cumulative distribution function (cdf), F(x), is defined as  For the Exponential distribution:

22. OS Fall ’ 02 The Exponential CDF

23. OS Fall ’ 02 Memoryless Property  The exponential is the only distribution with the property that  For modeling runtimes: the probability to run for additional b time units is the same regardless of how much the process has been running already in average

24. OS Fall ’ 02 Fat-tailed distribution  The real life workloads frequently do not fit the exponential distribution  Fat-tailed distributions:

25. OS Fall ’ 02 Pareto Distribution  The more you wait, the more additional time you should expect to wait  The longer a job has been running, the longer additional time it is expected to run

26. OS Fall ’ 02 Pareto dist. CDF

27. OS Fall ’ 02 Exponential vs Pareto

28. OS Fall ’ 02 Queuing Systems  Computing system can be viewed as a network of queues and servers  Shared queues are also possible CPU Disk A Disk B queue new jobs finished jobs

29. OS Fall ’ 02 The role of randomness  Arrival (departure) are random processes Deviations from the average are possible The deviation probabilities depend on the inter-arrival time distribution  Randomness makes you wait in queue Each job takes exactly 100ms to complete If jobs arrive each 100ms exactly, utilization is 100% But what if both these values are on average?

30. OS Fall ’ 02 Queuing analysis arriving jobs queue server departing jobs

31. OS Fall ’ 02 Little ’ s Law

32. OS Fall ’ 02 M/M/1 queue arriving jobs queue server departing jobs Both interarrival time and service time are exponentially distributed M stands for memoryless

33. OS Fall ’ 02 How average response time depends on utilization?  The job arrival and departure are approximated by Poisson processes  The distribution of the number of jobs in the system in the steady state is unique  Use queuing analysis to determine this distribution  Once it is known, can be found  Use the Little law to determine

34. OS Fall ’ 02 M/M/1 queue analysis 0132

35. OS Fall ’ 02

36. Response time (utilization)

37. OS Fall ’ 02 A Bank or a Supermarket? departing jobs departing jobs departing jobs departing jobs arriving jobs shared queue CPU1 CPU2 CPU3 CPU4 CPU1 CPU2 CPU3 CPU4 arriving jobs M/M/44 x M/M/1

38. OS Fall ’ 02 It is a Bank! M/M/4 4 x M/M/1

39. OS Fall ’ 02 Summary  What are the three main performance evaluation metrics?  What are the three main performance evaluation techniques?  What is the most important thing for performance evaluation?  Which workload models do you know?  What does make you to wait in queue?  How response time depends on utilization?

40. OS Fall ’ 02 To read more  Notes  Stallings, Appendix A  Raj Jain, The Art of Computer Performance Analysis

41. OS Fall ’ 02 Next:  Processes