Andy Wang CIS 5930 Computer Systems Performance Analysis Introduction Andy Wang CIS 5930 Computer Systems Performance Analysis

Overview of Analysis of Software Systems Introduction Common mistakes made in system software analysis And how to avoid them Selection of techniques and metrics

Introduction Why do we care? Why is it hard? What tools can we use to understand system performance?

Why Do We Care? Performance is almost always a key issue in software Everyone wants best possible performance Cost of achieving performance also key Reporting performance is necessary in many publication venues Both academic and industry

Importance of Performance in Research Performance is key in CS research A solution that doesn’t perform well isn’t a solution at all Must prove performance characteristics to a skeptical community

State of Performance Evaluation in the Field Generally regarded as poor Many systems have little performance data presented Measured by improper criteria Experiments poorly designed Results badly or incorrectly presented Replication not generally respected

What’s the Result? Can’t always trust what you read in a research paper Authors may have accidentally or intentionally misled you Overstating performance Reduce the power usage by disk seeks by 60% Hiding problems No mention of memory overhead Not answering the important questions

Where Does This Problem Come From? Mostly from ignorance of proper methods of measuring and presenting performance Abetted by reader’s ignorance of what questions they should be asking

But the Field Is Improving People are taking performance measurement more seriously Quality of published experiments is increasing Yours had better be of high quality, too Publishing is tough Be at the top of the heap of papers

What Do You Need To Know? Selection of proper experiment characteristics Proper performance measurement techniques Proper statistical techniques Proper data presentation techniques

Selecting Appropriate Experiment Characteristics Evaluation techniques Performance metrics Workloads

Evaluation Techniques What’s the best technique for getting an answer to a performance question? Modeling? Simulation? Experimentation?

The Role of Modeling and Simulation Measurement not always best tool Modeling might be better Simulation might be better But that’s not what this class is about We will discuss when to use those techniques, though

Performance Metrics The criteria used to evaluate the performance of a system E.g., response time, transactions per second, bandwidth delivered, etc. Choosing the proper metrics is key to a real understanding of system performance E.g., average response time, but how about dropped packets?

Workloads The requests users make on a system Without a proper workload, you aren’t measuring what users will experience Typical workloads: Types of queries Jobs submitted to an OS Messages sent through a protocol

Proper Performance Measurement Techniques You need at least two components to measure performance: 1. A load generator To apply a workload to the system 2. A monitor To find out what happened

Proper Statistical Techniques Most computer performance measurements not deterministic Most performance evaluations weigh effects of different alternatives How to separate meaningless variations from vital data in measurements? Requires proper statistical techniques

Minimizing Your Work Unless you design carefully, you’ll measure a lot more than you need to A careful design can save you from doing lots of measurements Should identify critical factors And determine smallest number of experiments that gives sufficiently accurate answer

Proper Data Presentation Techniques You’ve got pertinent, statistically accurate data that describes your system Now what? How to present it Honestly Clearly Convincingly

Why Is Performance Analysis Difficult? Because it’s an art - it’s not mechanical You can’t just apply a handful of principles and expect good results You’ve got to understand your system You’ve got to select your measurement techniques and tools properly You’ve got to be careful and honest

Bad Example: Ratio Game Throughput in transactions per second System Workload 1 Workload 2 Average A 20 10 15 B

Bad Example: Ratio Game Throughput normalized to system B Throughput normalized to system A System Workload 1 Workload 2 Average A 2 0.5 1.25 B 1 System Workload 1 Workload 2 Average A 1 B 0.5 2 1.25

Example Suppose you’ve built an OS for a special-purpose Internet browsing box How well does it perform? Indeed, how do you even begin to answer that question?

Starting on an Answer What’s the OS supposed to do? What demands will be put on it? What HW will it work with, and what are that hardware’s characteristics? What performance metrics are most important? Response time? Delivered bandwidth?

Some Common Mistakes in Performance Evaluation List is long (nearly infinite) We’ll cover the most common ones …and how to avoid them

No Goals If it’s not clear what you’re trying to find out, you won’t find out anything useful It’s very hard to design good general-purpose experiments and frameworks So know what you want to discover Think before you start

Biased Goals Don’t set out to show OUR system is better than THEIR system Doing so biases you towards using certain metrics, workloads, and techniques Which may not be the right ones Try not to let your prejudices dictate the way you measure a system Try to disprove your hypotheses

Unsystematic Approach Avoid scattershot approaches Work from a plan Follow through on the plan Otherwise, you’re likely to miss something And everything will take longer, in the end

Analysis Without Understanding the Problem Properly defining what you’re trying to do is a major component of the performance evaluation Don’t rush into gathering data till you understand what’s important This is the most common mistake in this class

Incorrect Performance Metrics If you don’t measure what’s important, your results don’t shed much light Example: instruction rates of CPUs with different architectures Avoid choosing the metric that’s easy to measure (but isn’t helpful) over the one that’s hard to measure (but captures the performance)

Unrepresentative Workload If workload isn’t like what normally happens, results aren’t useful E.g., for Web browser, it’s wrong to measure its performance against Just text pages Just pages stored on the local server

Wrong Evaluation Technique Not all performance problems are properly understood by measurement E.g., issues of scaling, or testing for rare cases Measurement is labor-intensive Sometimes hard to measure peak or unusual conditions Decide whether you should model/simulate before designing a measurement experiment

Overlooking Important Parameters Try to make a complete list of system/workload characteristics that affect performance Don’t just guess at a couple of interesting parameters Despite your best efforts, you may miss one anyway But the better you understand the system, the less likely you will

Ignoring Significant Factors Factor: parameter you vary Not all parameters equally important More factors  more experiments But don’t ignore significant ones Give weight to those that users can vary

Inappropriate Experiment Design Too few test runs Wrong values for parameters Interacting factors can complicate proper design of experiments

Inappropriate Level of Detail Be sure you’re investigating what’s important Examining at too high a level may oversimplify or miss important factors Going too low wastes time and may cause you to miss forest for trees

No Analysis Raw data isn’t too helpful Remember, final result is analysis that describes performance Preferably in compact form easily understood by others Especially for non-technical audiences Common mistake in this class (trying to satisfy page minimum in final report)

Erroneous Analysis Often based on misunderstanding of how statistics should be handled Or on not understanding transient effects in experiments Or in careless handling of the data Many other possible problems

No Sensitivity Analysis Rarely is one number or one curve fully descriptive of a system’s performance How different will things be if parameters are varied? Sensitivity analysis addresses this problem

Ignoring Errors in Input Particularly a problem when you have limited control over the parameters If you can’t measure an input parameter directly, need to understand any bias in indirect measurement

Improper Treatment of Outliers Sometimes particular data points are far outside range of all others Sometimes they’re purely statistical Sometimes they indicate a true, different behavior of system You must determine which are which

Assuming Static Systems A previous experiment may be useless if workload changes Or if key system parameters change Don’t rely blindly on old results E.g. Ousterhout’s old study of file servers Or old results suggesting superiority of diskless workstations

Ignoring Variability Often, showing only mean does not paint true picture of a system If variability is high, analysis and data presentation must make that clear

Overly Complex Analysis Choose simple explanations over complicated ones Choose simple experiments over complicated ones Choose simple explanations of phenomena Choose simple presentations of data

Improper Presentation of Results Real performance experiments are run to convince others Or help others make decisions If your results don’t convince or assist, you haven’t done any good Often, manner of presenting results makes the difference

Ignoring Social Aspects Raw graphs, tables, and numbers aren’t always enough for many people Need clear explanations Present in terms your audience understands Be especially careful if going against audience’s prejudices!

Omitting Assumptions and Limitations Tell your audience about assumptions and limitations Assumptions and limitations are usually necessary in real performance study But: Recognize them Be honest about them, Try to understand how they affect results

A Systematic Approach To Performance Evaluation 1. State goals and define the system 2. List services and outcomes 3. Select metrics 4. List parameters 5. Select factors to study 6. Select evaluation technique 7. Select workload

Systematic Approach, Cont. 8. Design experiments 9. Analyze and interpret data 10. Present results Usually, it is necessary to repeat some steps to get necessary answers

State Goals and Define the System Decide what you want to find out Put boundaries around what you’re going to consider This is a critical step, since everything afterwards is built on this foundation

List Services and Outcomes What services does system provide? What are possible results of requesting each service? Understanding these helps determine metrics

Select Metrics Metrics: criteria for determining performance Usually related to speed, accuracy, availability Studying inappropriate metrics makes a performance evaluation useless

List Parameters What system characteristics affect performance? System parameters Workload parameters Try to make it complete But expect you’ll add to list later

Select Factors To Study Factors: parameters that will be varied during study Best to choose factors most likely to be key to performance Also select levels for each factor Keep list and number of levels short Because it directly affects work involved

Select Evaluation Technique Analytic modeling? Simulation? Measurement? Or perhaps some combination Right choice depends on many issues

Select Workload List of service requests to be presented to system For measurement, usually workload is list of actual requests to system Must be representative of real workloads!

Design Experiments How to apply chosen workload? How to vary factors to their various levels? How to measure metrics? All with minimal effort Pay attention to interaction of factors

Analyze and Interpret Data Data gathered from measurement tend to have a random component Must extract the real information from the random noise Interpretation is key to getting value from experiment And is not at all mechanical Do this well in your project

Present Results Make them easy to understand For the broadest possible audience Focus on most important results Use graphics Give maximum information with minimum presentation Make sure you explain actual implications