1. Performance Evaluation
Operating Systems
Spring 2003
OS Spring’03

2. Performance evaluation
There are several approaches for implementing the same OS functionality
Different scheduling algorithms
Different memory management schemes
Performance evaluation deals with the question how to compare wellness of different approaches
Metrics, methods for evaluating metrics
OS Spring’03

3. Performance Metrics
What is the performance metric for the sorting algorithms?
Is something wrong with the following statement:
The complexity of my OS is O(n)?
This statement is inherently flawed
The reason: OS is a reactive program
OS Spring’03

4. Performance metrics Response time Throughput Utilization
Other metrics:
Mean Time Between Failures (MTBF)
Supportable load
OS Spring’03

5. Response time
The time interval between a user’s request and the system response
Response time, reaction time, turnaround time, etc.
Small response time is good:
For the user: waiting less
For the system: free to do other things
OS Spring’03

6. Throughput Number of work units done per time unit
Applications being run, files transferred, etc.
High throughput is good
For the system: was able to serve many clients
For the user: might imply worse service
OS Spring’03

7. Utilization Percentage of time the system is busy servicing clients
Important for expensive shared system
Less important (if at all)
for single user systems, for real time systems
Utilization and response time are interrelated
Very high utilization may negatively affect response time
OS Spring’03

8. Performance evaluation methods
Mathematical analysis
Based on a rigorous mathematical model
Simulation
Simulate the system operation (usually only small parts thereof)
Measurement
Implement the system in full and measure its performance directly
OS Spring’03

9. Analysis: Pros and Cons
Provides the best insight into the effects of different parameters and their interaction
Is it better to configure the system with one fast disk or with two slow disks?
Can be done before the system is built and takes a short time
Rarely accurate
Depends on host of simplifying assumptions
OS Spring’03

10. Simulation: Pros and Cons
Flexibility: full control of
Simulation model, parameters,
Level of detail
Disk: average seek time vs. acceleration and stabilization of the head
Can be done before the system is built
Simulation of a full system is infeasible
Simulation of the system parts does not take everything into account
OS Spring’03

11. Measurements: Pros and Cons
The most convincing
Effects of varying parameter values cannot (if at all) be easily isolated
Often confused with random changes in the environment
High cost:
Implement the system in full, buy hardware
OS Spring’03

12. The bottom line Simulation is the most widely used technique
Combination of techniques
Never trust the results produced by the single method
Validate with another one
E.g., simulation + analysis, simulation + measurements, etc.
OS Spring’03

13. Workload Workload is the sequence of things to do
Sequence of jobs submitted to the system
Arrival time, resources needed
File system: Sequence of I/O operations
Number of bytes to access
Workload is the input of the reactive system
The system performance depends on the workload
OS Spring’03

14. Workload analysis Workload modeling Recorded workload
Use past measurements to create a model
E.g., fit them into a distribution
Analysis, simulation, measurement
Recorded workload
Use past workload directly to drive evaluation
Simulation, measurement
OS Spring’03

15. Statistical characterization
Every workload item is sampled at random from the distribution of some random variable
Workload is characterized by a distribution
E.g., take all possible job times and fit them to a distribution
OS Spring’03

16. The Exponential Distribution
A lot of low values and a few high values
The distributions of salaries, lifetimes, and waiting times are often fit the exponential distribution
The distribution of
Job runtimes
Job inter-arrival times
File sizes
OS Spring’03

17. Exponential Probability Density Function (pdf)
If X has an exponential distribution with parameter , then its probability density function is given by:
where
OS Spring’03

18. The Exponential Distribution
OS Spring’03

19. Mean and Variance
If X~Exponential( ), then its mean and variance are given by:
OS Spring’03

20. Exponential Probabilities
f(x)
f(x) x x a a
OS Spring’03

21. Memoryless Property
The exponential is the only distribution with the property that
For modeling runtimes: the probability to run for additional b time units is the same regardless of how much the process has been running already
in average
OS Spring’03

22. Fat-tailed distribution
The real life workloads frequently do not fit the exponential distribution
Fat-tailed distributions:
OS Spring’03

23. Pareto Distribution Mean is unbounded
The more you wait, the more additional time you should expect to wait
The longer a job has been running, the longer additional time it is expected to run
OS Spring’03

24. Exponential vs. Pareto
The mean additional time to wait is determined by the shape of the tail
The fatter tail, the more additional time to wait
For exp.: the tail shape is the same regardless of how much we have waited already=>
The mean additional time stays the same
For Pareto: The more we wait, the fatter tail becomes
The more we wait, the more additional time we will wait
OS Spring’03

25. Exp. vs. Pareto: Focus on tail
OS Spring’03

26. Queuing Systems
queue
Disk A
queue
Disk B
new
jobs
finished
jobs
CPU
queue
Computing system can be viewed as a network of queues and servers
OS Spring’03

27. The role of randomness Arrival (departure) are random processes
Deviations from the average are possible
The deviation probabilities depend on the inter-arrival time distribution
Randomness makes you wait in queue
Each job takes exactly 100ms to complete
If jobs arrive each 100ms exactly, utilization is 100%
But what if both these values are on average?
OS Spring’03

28. Queuing analysis server queue departing arriving jobs jobs
OS Spring’03

29. Little’s Law
OS Spring’03

30. How response time depends on utilization?
Write the average number of jobs as a function of arrival and service rates
Queuing analysis
Substitute it to the Little’s law
OS Spring’03

31. M/M/1 queue analysis 1 3 2
OS Spring’03

32. OS Spring’03

33. OS Spring’03

34. OS Spring’03

35. OS Spring’03

36. Response time (utilization)
OS Spring’03

37. Summary What are the three main performance evaluation metrics?
What are the three main performance evaluation techniques?
What is the most important thing for performance evaluation?
Which workload models do you know?
What does make you to wait in queue?
How response time depends on utilization?
OS Spring’03

38. To read more Notes Stallings, Appendix A
Raj Jain, The Art of Computer Performance Analysis
OS Spring’03

39. Next:
Processes
OS Spring’03