1. Andy Wang CIS 5930 Computer Systems Performance Analysis
Introduction
Andy Wang
CIS 5930
Computer Systems
Performance Analysis

2. Overview of Analysis of Software Systems
Introduction
Common mistakes made in system software analysis
And how to avoid them
Selection of techniques and metrics

3. Introduction Why do we care? Why is it hard?
What tools can we use to understand system performance?

4. Why Do We Care? Performance is almost always a key issue in software
Everyone wants best possible performance
Cost of achieving performance also key
Reporting performance is necessary in many publication venues
Both academic and industry

5. Importance of Performance in Research
Performance is key in CS research
A solution that doesn’t perform well isn’t a solution at all
Must prove performance characteristics to a skeptical community

6. State of Performance Evaluation in the Field
Generally regarded as poor
Many systems have little performance data presented
Measured by improper criteria
Experiments poorly designed
Results badly or incorrectly presented
Replication not generally respected

7. What’s the Result?
Can’t always trust what you read in a research paper
Authors may have accidentally or intentionally misled you
Overstating performance
Reduce the power usage by disk seeks by 60%
Hiding problems
No mention of memory overhead
Not answering the important questions

8. Where Does This Problem Come From?
Mostly from ignorance of proper methods of measuring and presenting performance
Abetted by reader’s ignorance of what questions they should be asking

9. But the Field Is Improving
People are taking performance measurement more seriously
Quality of published experiments is increasing
Yours had better be of high quality, too
Publishing is tough
Be at the top of the heap of papers

10. What Do You Need To Know?
Selection of proper experiment characteristics
Proper performance measurement techniques
Proper statistical techniques
Proper data presentation techniques

11. Selecting Appropriate Experiment Characteristics
Evaluation techniques
Performance metrics
Workloads

12. Evaluation Techniques
What’s the best technique for getting an answer to a performance question?
Modeling?
Simulation?
Experimentation?

13. The Role of Modeling and Simulation
Measurement not always best tool
Modeling might be better
Simulation might be better
But that’s not what this class is about
We will discuss when to use those techniques, though

14. Performance Metrics
The criteria used to evaluate the performance of a system
E.g., response time, transactions per second, bandwidth delivered, etc.
Choosing the proper metrics is key to a real understanding of system performance
E.g., average response time, but how about dropped packets?

15. Workloads The requests users make on a system
Without a proper workload, you aren’t measuring what users will experience
Typical workloads:
Types of queries
Jobs submitted to an OS
Messages sent through a protocol

16. Proper Performance Measurement Techniques
You need at least two components to measure performance:
1. A load generator
To apply a workload to the system
2. A monitor
To find out what happened

17. Proper Statistical Techniques
Most computer performance measurements not deterministic
Most performance evaluations weigh effects of different alternatives
How to separate meaningless variations from vital data in measurements?
Requires proper statistical techniques

18. Minimizing Your Work
Unless you design carefully, you’ll measure a lot more than you need to
A careful design can save you from doing lots of measurements
Should identify critical factors
And determine smallest number of experiments that gives sufficiently accurate answer

19. Proper Data Presentation Techniques
You’ve got pertinent, statistically accurate data that describes your system
Now what?
How to present it
Honestly
Clearly
Convincingly

20. Why Is Performance Analysis Difficult?
Because it’s an art - it’s not mechanical
You can’t just apply a handful of principles and expect good results
You’ve got to understand your system
You’ve got to select your measurement techniques and tools properly
You’ve got to be careful and honest

21. Bad Example: Ratio Game
Throughput in transactions per second
System
Workload 1
Workload 2
Average A 20 10 15 B

22. Bad Example: Ratio Game
Throughput normalized to system B
Throughput normalized to system A
System
Workload 1
Workload 2
Average A 2
0.5
1.25 B 1
System
Workload 1
Workload 2
Average A 1 B
0.5 2
1.25

23. Example
Suppose you’ve built an OS for a special-purpose Internet browsing box
How well does it perform?
Indeed, how do you even begin to answer that question?

24. Starting on an Answer What’s the OS supposed to do?
What demands will be put on it?
What HW will it work with, and what are that hardware’s characteristics?
What performance metrics are most important?
Response time?
Delivered bandwidth?

25. Some Common Mistakes in Performance Evaluation
List is long (nearly infinite)
We’ll cover the most common ones
…and how to avoid them

26. No Goals
If it’s not clear what you’re trying to find out, you won’t find out anything useful
It’s very hard to design good general-purpose experiments and frameworks
So know what you want to discover
Think before you start

27. Biased Goals
Don’t set out to show OUR system is better than THEIR system
Doing so biases you towards using certain metrics, workloads, and techniques
Which may not be the right ones
Try not to let your prejudices dictate the way you measure a system
Try to disprove your hypotheses

28. Unsystematic Approach
Avoid scattershot approaches
Work from a plan
Follow through on the plan
Otherwise, you’re likely to miss something
And everything will take longer, in the end

29. Analysis Without Understanding the Problem
Properly defining what you’re trying to do is a major component of the performance evaluation
Don’t rush into gathering data till you understand what’s important
This is the most common mistake in this class

30. Incorrect Performance Metrics
If you don’t measure what’s important, your results don’t shed much light
Example: instruction rates of CPUs with different architectures
Avoid choosing the metric that’s easy to measure (but isn’t helpful) over the one that’s hard to measure (but captures the performance)

31. Unrepresentative Workload
If workload isn’t like what normally happens, results aren’t useful
E.g., for Web browser, it’s wrong to measure its performance against
Just text pages
Just pages stored on the local server

32. Wrong Evaluation Technique
Not all performance problems are properly understood by measurement
E.g., issues of scaling, or testing for rare cases
Measurement is labor-intensive
Sometimes hard to measure peak or unusual conditions
Decide whether you should model/simulate before designing a measurement experiment

33. Overlooking Important Parameters
Try to make a complete list of system/workload characteristics that affect performance
Don’t just guess at a couple of interesting parameters
Despite your best efforts, you may miss one anyway
But the better you understand the system, the less likely you will

34. Ignoring Significant Factors
Factor: parameter you vary
Not all parameters equally important
More factors  more experiments
But don’t ignore significant ones
Give weight to those that users can vary

35. Inappropriate Experiment Design
Too few test runs
Wrong values for parameters
Interacting factors can complicate proper design of experiments

36. Inappropriate Level of Detail
Be sure you’re investigating what’s important
Examining at too high a level may oversimplify or miss important factors
Going too low wastes time and may cause you to miss forest for trees

37. No Analysis Raw data isn’t too helpful
Remember, final result is analysis that describes performance
Preferably in compact form easily understood by others
Especially for non-technical audiences
Common mistake in this class (trying to satisfy page minimum in final report)

38. Erroneous Analysis
Often based on misunderstanding of how statistics should be handled
Or on not understanding transient effects in experiments
Or in careless handling of the data
Many other possible problems

39. No Sensitivity Analysis
Rarely is one number or one curve fully descriptive of a system’s performance
How different will things be if parameters are varied?
Sensitivity analysis addresses this problem

40. Ignoring Errors in Input
Particularly a problem when you have limited control over the parameters
If you can’t measure an input parameter directly, need to understand any bias in indirect measurement

41. Improper Treatment of Outliers
Sometimes particular data points are far outside range of all others
Sometimes they’re purely statistical
Sometimes they indicate a true, different behavior of system
You must determine which are which

42. Assuming Static Systems
A previous experiment may be useless if workload changes
Or if key system parameters change
Don’t rely blindly on old results
E.g. Ousterhout’s old study of file servers
Or old results suggesting superiority of diskless workstations

43. Ignoring Variability
Often, showing only mean does not paint true picture of a system
If variability is high, analysis and data presentation must make that clear

44. Overly Complex Analysis
Choose simple explanations over complicated ones
Choose simple experiments over complicated ones
Choose simple explanations of phenomena
Choose simple presentations of data

45. Improper Presentation of Results
Real performance experiments are run to convince others
Or help others make decisions
If your results don’t convince or assist, you haven’t done any good
Often, manner of presenting results makes the difference

46. Ignoring Social Aspects
Raw graphs, tables, and numbers aren’t always enough for many people
Need clear explanations
Present in terms your audience understands
Be especially careful if going against audience’s prejudices!

47. Omitting Assumptions and Limitations
Tell your audience about assumptions and limitations
Assumptions and limitations are usually necessary in real performance study
But:
Recognize them
Be honest about them,
Try to understand how they affect results

48. A Systematic Approach To Performance Evaluation
1. State goals and define the system
2. List services and outcomes
3. Select metrics
4. List parameters
5. Select factors to study
6. Select evaluation technique
7. Select workload

49. Systematic Approach, Cont.
8. Design experiments
9. Analyze and interpret data
10. Present results
Usually, it is necessary to repeat some steps to get necessary answers

50. State Goals and Define the System
Decide what you want to find out
Put boundaries around what you’re going to consider
This is a critical step, since everything afterwards is built on this foundation

51. List Services and Outcomes
What services does system provide?
What are possible results of requesting each service?
Understanding these helps determine metrics

52. Select Metrics Metrics: criteria for determining performance
Usually related to speed, accuracy, availability
Studying inappropriate metrics makes a performance evaluation useless

53. List Parameters What system characteristics affect performance?
System parameters
Workload parameters
Try to make it complete
But expect you’ll add to list later

54. Select Factors To Study
Factors: parameters that will be varied during study
Best to choose factors most likely to be key to performance
Also select levels for each factor
Keep list and number of levels short
Because it directly affects work involved

55. Select Evaluation Technique
Analytic modeling?
Simulation?
Measurement?
Or perhaps some combination
Right choice depends on many issues

56. Select Workload List of service requests to be presented to system
For measurement, usually workload is list of actual requests to system
Must be representative of real workloads!

57. Design Experiments How to apply chosen workload?
How to vary factors to their various levels?
How to measure metrics?
All with minimal effort
Pay attention to interaction of factors

58. Analyze and Interpret Data
Data gathered from measurement tend to have a random component
Must extract the real information from the random noise
Interpretation is key to getting value from experiment
And is not at all mechanical
Do this well in your project

59. Present Results Make them easy to understand
For the broadest possible audience
Focus on most important results
Use graphics
Give maximum information with minimum presentation
Make sure you explain actual implications