OS Fall ’ 02 Performance Evaluation Operating Systems Fall 2002

OS Fall ’ 02 Performance evaluation  There are several approaches for implementing the same OS functionality Different scheduling algorithms Different memory management schemes  Performance evaluation deals with the question how to compare wellness of different approaches Metrics, methods for evaluating metrics

OS Fall ’ 02 Performance Metrics  Is something wrong with the following statement: The complexity of my OS is O(n)?  This statement is inherently flawed The reason: OS is a reactive program  What is the performance metric for the sorting algorithms?

OS Fall ’ 02 Performance metrics  Response time  Throughput  Utilization  Other metrics: Mean Time Between Failures (MTBF) Supportable load

OS Fall ’ 02 Response time  The time interval between a user ’ s request and the system response Response time, reaction time, turnaround time, etc.  Small response time is good: For the user: waiting less For the system: free to do other things

OS Fall ’ 02 Throughput  Number of work units done per time unit Applications being run, files transferred, etc.  High throughput is good For the system: was able to serve many clients For the user: might imply worse service

OS Fall ’ 02 Utilization  Percentage of time the system is busy servicing clients Important for expensive shared system Less important (if at all)  for single user systems, for real time systems  Utilization and response time are interrelated Very high utilization may negatively affect response time

OS Fall ’ 02 Performance evaluation methods  Mathematical analysis Based on a rigorous mathematical model  Simulation Simulate the system operation (usually only small parts thereof)  Measurement Implement the system in full and measure its performance directly

OS Fall ’ 02 Analysis: Pros and Cons + Provides the best insight into the effects of different parameters and their interaction Is it better to configure the system with one fast disk or with two slow disks? + Can be done before the system is built and takes a short time - Rarely accurate Depends on host of simplifying assumptions

OS Fall ’ 02 Simulation: Pros and Cons + Flexibility: full control of Simulation model, parameters, Level of detail  Disk: average seek time vs. acceleration and stabilization of the head + Can be done before the system is built - Simulation of a full system is infeasible - Simulation of the system parts does not take everything into account

OS Fall ’ 02 Measurements: Pros and Cons + The most convincing - Effects of varying parameter values cannot (if at all) be easily isolated Often confused with random changes in the environment - High cost: Implement the system in full, buy hardware

OS Fall ’ 02 The bottom line  Simulation is the most widely used technique  Combination of techniques Never trust the results produced by the single method  Validate with another one  E.g., simulation + analysis, simulation + measurements, etc.

OS Fall ’ 02 Workload  Workload is the sequence of things to do Sequence of jobs submitted to the system  Arrival time, resources needed File system: Sequence of I/O operations  Number of bytes to access  Workload is the input of the reactive system The system performance depends on the workload

OS Fall ’ 02 Workload analysis  Workload modeling Use past measurements to create a model  E.g., fit them into a distribution Analysis, simulation, measurement  Recorded workload Use past workload directly to drive evaluation Simulation, measurement

OS Fall ’ 02 Statistical characterization  Every workload item is sampled at random from the distribution of some random variable Workload is characterized by a distribution E.g., take all possible job times and fit them to a distribution

OS Fall ’ 02 The Exponential Distribution  A lot of low values and a few high values  The distributions of salaries, lifetimes, and waiting times are often fit the exponential distribution  The distribution of Job runtimes Job inter-arrival times File sizes

OS Fall ’ 02 Exponential Probability Density Function (pdf)  If X has an exponential distribution with parameter, then its probability density function is given by: where

OS Fall ’ 02 The Exponential Distribution

OS Fall ’ 02 Mean and Variance  If X~Exponential( ), then its mean and variance are given by:

OS Fall ’ 02 Exponential Probabilities a x f(x) a x 00

OS Fall ’ 02 Cumulative Distribution Function  The cumulative distribution function (cdf), F(x), is defined as  For the Exponential distribution:

OS Fall ’ 02 The Exponential CDF

OS Fall ’ 02 Memoryless Property  The exponential is the only distribution with the property that  For modeling runtimes: the probability to run for additional b time units is the same regardless of how much the process has been running already in average

OS Fall ’ 02 Fat-tailed distribution  The real life workloads frequently do not fit the exponential distribution  Fat-tailed distributions:

OS Fall ’ 02 Pareto Distribution  The more you wait, the more additional time you should expect to wait  The longer a job has been running, the longer additional time it is expected to run

OS Fall ’ 02 Pareto dist. CDF

OS Fall ’ 02 Exponential vs Pareto

OS Fall ’ 02 Queuing Systems  Computing system can be viewed as a network of queues and servers  Shared queues are also possible CPU Disk A Disk B queue new jobs finished jobs

OS Fall ’ 02 The role of randomness  Arrival (departure) are random processes Deviations from the average are possible The deviation probabilities depend on the inter-arrival time distribution  Randomness makes you wait in queue Each job takes exactly 100ms to complete If jobs arrive each 100ms exactly, utilization is 100% But what if both these values are on average?

OS Fall ’ 02 Queuing analysis arriving jobs queue server departing jobs

OS Fall ’ 02 Little ’ s Law

OS Fall ’ 02 M/M/1 queue arriving jobs queue server departing jobs Both interarrival time and service time are exponentially distributed M stands for memoryless

OS Fall ’ 02 How average response time depends on utilization?  The job arrival and departure are approximated by Poisson processes  The distribution of the number of jobs in the system in the steady state is unique  Use queuing analysis to determine this distribution  Once it is known, can be found  Use the Little law to determine

OS Fall ’ 02 M/M/1 queue analysis 0132

OS Fall ’ 02

Response time (utilization)

OS Fall ’ 02 A Bank or a Supermarket? departing jobs departing jobs departing jobs departing jobs arriving jobs shared queue CPU1 CPU2 CPU3 CPU4 CPU1 CPU2 CPU3 CPU4 arriving jobs M/M/44 x M/M/1

OS Fall ’ 02 It is a Bank! M/M/4 4 x M/M/1

OS Fall ’ 02 Summary  What are the three main performance evaluation metrics?  What are the three main performance evaluation techniques?  What is the most important thing for performance evaluation?  Which workload models do you know?  What does make you to wait in queue?  How response time depends on utilization?

OS Fall ’ 02 To read more  Notes  Stallings, Appendix A  Raj Jain, The Art of Computer Performance Analysis

OS Fall ’ 02 Next:  Processes