1. CPE 619: Modeling and Analysis of Computer and Communications Systems Aleksandar Milenković The LaCASA Laboratory Electrical and Computer Engineering Department The University of Alabama in Huntsville http://www.ece.uah.edu/~milenka http://www.ece.uah.edu/~lacasa

2. 2 Topics Background About the Course

3. 3 Professor Background Dr. Aleksandar Milenković Research interests LaCASA laboratory: www.ece.uah.edu/~lacasawww.ece.uah.edu/~lacasa Computer architecture: hardware/software structures for secure, cost-effective, and performance-effective computation and communication Performance analysis and evaluation Low-power deeply embedded systems (sensor networks) Teaching interests CPE 631: Advanced Computer Systems Architecture CPE 619: Modeling and Analysis of Computer and Communication Systems CPE 527: VLSI Design CPE 323: Introduction to Embedded Computer Systems

4. 4 Goals of This Course Comprehensive course on performance analysis Includes measurement, statistical modeling, experimental design, simulation, and queuing theory How to avoid common mistakes in performance analysis Graduate course: (Advanced Topics)  Lot of independent reading and writing  Project/Survey paper (Research techniques)

5. 5 Syllabus Course Web page: http://www.ece.uah.edu/~milenka/cpe619-08S Office hours: MW 2:15 PM – 3:15 PM Email: milenka at ece.uah.edu

6. 6 Text Books Required Raj Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley and Sons, Inc., New York, NY, 1991. ISBN:0471503363The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling Other Edward D. Lazowska, John Zahorjan, G. Scott Graham, and Kenneth C. Sevcik. Computer System Analysis Using Queueing Network Models. NOTE: Available for free on the WEB at http://www.cs.washington.edu/homes/lazowska/qsp/ David J. Lilja. Measuring Computer Performance: A Practitioner's Guide, Cambridge University Press, New York, NY, 2000.Measuring Computer Performance: A Practitioner's Guide

7. 7 Objectives: What You Will Learn Specifying performance requirements Evaluating design alternatives Comparing two or more systems Determining the optimal value of a parameter (system tuning) Finding the performance bottleneck (bottleneck identification) Characterizing the load on the system (workload characterization) Determining the number and sizes of components (capacity planning) Predicting the performance at future loads (forecasting)

8. 8 Basic Terms System: Any collection of hardware, software, and firmware Metrics: Criteria used to evaluate the performance of the system components Workloads: The requests made by the users of the system

9. 9 Main Parts of the Course Part I: An Overview of Performance Evaluation Part II: Measurement Techniques and Tools Part III: Probability Theory and Statistics Part IV: Experimental Design and Analysis Part V: Simulation Part VI: Queuing Theory

10. 10 Part I: An Overview of Performance Evaluation Introduction Common Mistakes and How To Avoid Them Selection of Techniques and Metrics

11. 11 Part II: Measurement Techniques and Tools Types of Workloads Popular Benchmarks The Art of Workload Selection Workload Characterization Techniques Monitors Accounting Logs Monitoring Distributed Systems Load Drivers Capacity Planning The Art of Data Presentation Ratio Games

12. 12 Part III: Probability Theory and Statistics Probability and Statistics Concepts Four Important Distributions Summarizing Measured Data By a Single Number Summarizing The Variability Of Measured Data Graphical Methods to Determine Distributions of Measured Data Sample Statistics Confidence Interval Comparing Two Alternatives Measures of Relationship Simple Linear Regression Models Multiple Linear Regression Models Other Regression Models

13. 13 Part IV: Experimental Design and Analysis Introduction to Experimental Design 2 k Factorial Designs 2 k r Factorial Designs with Replications 2 k-p Fractional Factorial Designs One Factor Experiments Two Factors Full Factorial Design without Replications Two Factors Full Factorial Design with Replications General Full Factorial Designs With k Factors

14. 14 Part V: Simulation Introduction to Simulation Types of Simulations Model Verification and Validation Analysis of Simulation Results Random-Number Generation Testing Random-Number Generators Random-Variate Generation Commonly Used Distributions

15. 15 Part VI: Queuing Theory Introduction to Queueing Theory Analysis of A Single Queue Queuing Networks Operational Laws Mean Value Analysis and Related Techniques Convolution Algorithm Advanced Techniques

16. 16 Grading Policy Prerequisites MA 585 or EE 500 (soft requirement) General knowledge about computer systems Interest in performance evaluation Grading Homeworks 20% Midterm Exam25% Final Exam25% Project25% Class participation 5%

17. 17 More on Projects Goal: Provide an insight (or information) not obvious before the project Two project types A survey paper on a performance topic A real case study on performance of a system you are already working on

18. 18 Other Course Policies See the course web site

19. Introduction

20. 20 Outline  Objectives  The Art  Common Mistakes  Systematic Approach  Case Study

21. 21 Objectives (1 of 6) Select appropriate evaluation techniques, performance metrics and workloads for a system Techniques: measurement, simulation, analytic modeling Metrics: criteria to study performance (ex: response time) Workloads: requests by users/applications to the system Example: What performance metrics should you use for the following systems? a) Two disk drives b) Two transactions processing systems c) Two packet retransmission algorithms

22. 22 Objectives (2 of 6) Conduct performance measurements correctly Need two tools Load generator – a tool to load the system Monitor – a tool to measure the results Example: Which type of monitor (software and hardware) would be more suitable for measuring each of the following quantities? a) Number of instructions executed by a processor b) Degree of multiprogramming on a timesharing system c) Response time of packets on a network

23. 23 Objectives (3 of 6) Use proper statistical techniques to compare several alternatives Find the best among a number of alternatives One run of workload often not sufficient Many non-deterministic computer events that effect performance Comparing average of several runs may also not lead to correct results Especially if variance is high Example: Packets lost on a link. Which link is better? File SizeLink ALink B 1000510 120073 130030 5001

24. 24 Objectives (4 of 6) Design measurement and simulation experiments to provide the most information with the least effort Often many factors that affect performance. Separate out the effects that individual factors. Example: The performance of a system depends upon three factors: A) garbage collection technique: G1, G2, or none B) type of workload: editing, compiling, AI C) type of CPU: P2, P4, Sparc How many experiments are needed? How can the performance of each factor be estimated?

25. 25 Objectives (5 of 6) Perform simulations correctly Select correct language, seeds for random numbers, length of simulation run, and analysis Before all of that, may need to validate simulator Example: To compare the performance of two cache replacement algorithms: A) What type of simulation model should be used? B) How long should the simulation be run? C) What can be done to get the same accuracy with a shorter run? D) How can one decide if the random-number generator in the simulation is a good generator?

26. 26 Objectives (6 of 6) Use simple queuing models to analyze the performance of systems Queuing models are commonly used for analytical modeling of computer systems Often can model computer systems by service rate and arrival rate of load Multiple servers Multiple queues Example: The average response time of a database system is 3 seconds. During a 1-minute observation interval, the idle time on the system was 10 seconds. Using a queuing model for the system, determine the following: System utilization, average service time per query, the number of queries completed during observation, average number of jobs in the system, …

27. 27 Outline  Objectives  The Art  Common Mistakes  Systematic Approach  Case Study

28. 28 The Art of Performance Evaluation Evaluation cannot be produced mechanically Requires intimate knowledge of system Careful selection of methodology, workload, tools No one correct answer as two performance analysts may choose different metrics or workloads Like art, there are techniques to learn how to use them when to apply them

29. 29 Example: Comparing Two Systems Two systems, two workloads, measure transactions per second Which is better? SystemWorkload 1Workload 2 A2010 B 20

30. 30 Example: Comparing Two Systems Two systems, two workloads, measure transactions per second They are equally good! … but is A better than B? SystemWorkload 1Workload 2Average A201015 B102015

31. 31 The Ratio Game Take system B as the base A is better! … but is B better than A? SystemWorkload 1Workload 2Average A20.51.25 B111

32. 32 The Ratio Game Take system A as the base B is better!? SystemWorkload 1Workload 2Average A111 B0.521.25

33. 33 Outline  Objectives  The Art  Common Mistakes  Systematic Approach  Case Study

34. 34 Common Mistakes (1-4) 1. Undefined Goals (Don’t shoot and then draw target) There is no such thing as a general model Describe goals and then design experiments 2. Biased Goals (Performance analysis is like a jury) Don’t show YOUR system better than HERS 3. Unsystematic Approach Arbitrary selection of system parameters, factors, metrics, … will lead to inaccurate conclusions 4. Analysis without Understanding (“A problem well-stated is half solved”) Don’t rush to modeling before defining a problem

35. 35 Common Mistakes (5-8) 5. Incorrect Performance Metrics E.g., MIPS 6. Unrepresentative Workload Wrong workload will lead to inaccurate conclusions 7. Wrong Evaluation Technique (Don’t have a hammer and see everything as a nail) Use most appropriate: model, simulation, measurement 8. Overlooking Important Parameters Start from a complete list of system and workload parameters that affect the performance

36. 36 Common Mistakes (9-12) 9. Ignoring Significant Factors Parameters that are varied are called factors; others are fixed Identify parameters that make significant impact on performance when varied 10. Inappropriate Experimental Design Relates to the number of measurement or simulation experiments to be conducted 11. Inappropriate Level of Detail Can have too much! Ex: modeling disk Can have too little! Ex: analytic model for congested router 12. No Analysis Having a measurement expert is desirable but not enough Expertise in analyzing results is crucial

37. 37 Common Mistakes (13-16) 13. Erroneous Analysis E.g., take averages on too short simulations 14. No Sensitivity Analysis Analysis is evidence and not fact Need to determine how sensitive results are to settings 15. Ignoring Errors in Input Often parameters of interest cannot be measured; Instead, they are estimated using other variables Adjust the level of confidence on the model output 16. Improper Treatment of Outliers Outliers are values that are too high or too low compared to a majority of values If possible in real systems or workloads, do not ignore them

38. 38 Common Mistakes (17-20) 17. Assuming No Change in the Future Workload may change in the future 18. Ignoring Variability If variability is high, the mean performance alone may be misleading 19. Too Complex Analysis A simpler and easier to explain analysis should be preferred 20. Improper Presentation of Results It is not the number of graphs, but the number of graphs that help make decisions

39. 39 Common Mistakes (21-22) 21. Ignoring Social Aspects Writing and speaking are social skills 22. Omitting Assumptions and Limitations E.g.: may assume most traffic TCP, whereas some links may have significant UDP traffic May lead to applying results where assumptions do not hold

40. 40 Checklist for Avoiding Common Mistakes in Performance Evaluation 1. Is the system correctly defined and the goals are clearly stated? 2. Are the goals stated in an unbiased manner? 3. Have all the steps of the analysis followed systematically? 4. Is the problem clearly understood before analyzing it? 5. Are the performance metrics relevant for this problem? 6. Is the workload correct for this problem? 7. Is the evaluation technique appropriate? 8. Is the list of parameters that affect performance complete? 9. Have all parameters that affect performance been chosen as factors to be varied? 10. Is the experimental design efficient in terms of time and results? 11. Is the level of detail proper? 12. Is the measured data presented with analysis and interpretation? 13. Is the analysis statistically correct? 14. Has the sensitivity analysis been done? 15. Would errors in the input cause an insignificant change in the results? 16. Have the outliers in the input or the output been treated properly? 17. Have the future changes in the system and workload been modeled? 18. Has the variance of input been taken into account? 19. Has the variance of the results been analyzed? 20. Is the analysis easy to explain? 21. Is the presentation style suitable for its audience? 22. Have the results been presented graphically as much as possible? 23. Are the assumptions and limitations of the analysis clearly documented?

41. 41 Outline  Objectives  The Art  Common Mistakes  Systematic Approach  Case Study

42. 42 A Systematic Approach 1.State goals and define boundaries 2.List services and outcomes 3.Select performance metrics 4.List system and workload parameters 5.Select factors and values 6.Select evaluation techniques 7.Select workload 8.Design experiments 9.Analyze and interpret the data 10.Present the results. Repeat.

43. 43 State Goals and Define Boundaries Just “measuring performance” or “seeing how it works” is too broad E.g.: goal is to decide which ISP provides better throughput Definition of system may depend upon goals E.g.: if measuring CPU instruction speed, system may include CPU + cache E.g.: if measuring response time, system may include CPU + memory + … + OS + user workload

44. 44 List Services and Outcomes List services provided by the system E.g., a computer network allows users to send packets to specified destinations E.g., a database system responds to queries E.g., a processor performs a number of tasks A user request for any of these services results in a number of possible outcomes (desirable or not) E.g., a database system may answer correctly, incorrectly (due to inconsistent updates), or not at all (due to deadlocks)

45. 45 Select Metrics Criteria to compare performance In general, related to speed, accuracy and/or availability of system services E.g.: network performance Speed: throughput and delay Accuracy: error rate Availability: data packets sent do arrive E.g.: processor performance Speed: time to execute instructions

46. 46 List Parameters List all parameters that affect performance System parameters (hardware and software) E.g.: CPU type, OS type, … Workload parameters E.g.: Number of users, type of requests List may not be initially complete, so have a working list and let grow as progress

47. 47 Select Factors to Study Divide parameters into those that are to be studied and those that are not E.g.: may vary CPU type but fix OS type E.g.: may fix packet size but vary number of connections Select appropriate levels for each factor Want typical and ones with potentially high impact For workload often smaller (1/2 or 1/10 th ) and larger (2x or 10x) range Start small or number can quickly overcome available resources!

48. 48 Select Evaluation Technique Depends upon time, resources, and desired level of accuracy Analytic modeling Quick, less accurate Simulation Medium effort, medium accuracy Measurement Typical most effort, most accurate Note, above are all typical but can be reversed in some cases!

49. 49 Select Workload Set of service requests to system Depends upon measurement technique Analytic model may have probability of various requests Simulation may have trace of requests from real system Measurement may have scripts impose transactions Should be representative of real life

50. 50 Design Experiments Want to maximize results with minimal effort Phase 1: Many factors, few levels See which factors matter Phase 2: Few factors, more levels See where the range of impact for the factors is

51. 51 Analyze and Interpret Data Compare alternatives Take into account variability of results Statistical techniques Interpret results The analysis does not provide a conclusion Different analysts may come to different conclusions

52. 52 Present Results Make it easily understood Graphs Disseminate (entire methodology!) "The job of a scientist is not merely to see: it is to see, understand, and communicate. Leave out any of these phases, and you're not doing science. If you don't see, but you do understand and communicate, you're a prophet, not a scientist. If you don't understand, but you do see and communicate, you're a reporter, not a scientist. If you don't communicate, but you do see and understand, you're a mystic, not a scientist."

53. 53 Outline  Objectives  The Art  Common Mistakes  Systematic Approach  Case Study

54. 54 Case Study Consider remote pipes (rpipe) versus remote procedure calls (rpc) rpc is like procedure call but procedure is handled on remote server Client caller blocks until return rpipe is like pipe but server gets output on remote machine Client process can continue, non-blocking Results are returned asynchronously Goal: study the performance of applications using rpipes to similar applications using rpcs

55. 55 System Definition Client and Server and Network Key component is “channel”, either a rpipe or an rpc Only the subset of the client and server that handle channel are part of the system Client Network Server - Try to minimize effect of components outside system

56. 56 Services There are a variety of services that can happen over a rpipe or rpc Choose data transfer as a common one, with data being a typical result of most client-server interactions Classify amount of data as either large or small Thus, two services: Small data transfer Large data transfer

57. 57 Metrics Limit metrics to correct operation only (no failure or errors) Study service rate and resources consumed Performance metrics A) elapsed time per call B) maximum call rate per unit time C) Local CPU time per call D) Remote CPU time per call E) Number of bytes sent per call

58. 58 Parameters Speed of CPUs Local Remote Network Speed Reliability (retrans) Operating system overhead For interfacing with channels For interfacing with network Time between calls Number and sizes of parameters of results Type of channel rpc Rpipe Other loads On CPUs On network SystemWorkload

59. 59 Key Factors A) Type of channel rpipe or rpc B) Speed of network Choose short (LAN) and across country (WAN) C) Size of parameters Small or larger D) Number of calls 11 values: 8, 16, 32 …1024 E) All other parameters are fixed (Note, try to run during “light” network load)

60. 60 Evaluation Technique Since there are prototypes, use measurement Use analytic modeling based on measured data for values outside the scope of the experiments conducted

61. 61 Workload Synthetic program generated specified channel requests Will also monitor resources consumed and log results Use “null” channel requests to get baseline resources consumed by logging Heisenberg uncertainty principle in physics: “the measurement of position necessarily disturbs a particle's momentum, and vice versa—i.e., that the uncertainty principle is a manifestation of the observer effect”observer effect

62. 62 Experimental Design Full factorial (all possible combinations of factors) 2 channels, 2 network speeds, 2 sizes, 11 numbers of calls  2 x 2 x 2 x 11 = 88 experiments

63. 63 Data Analysis Analysis of variance will be used to quantify the first three factors Are they different? Regression will be used to quantify the effects of n consecutive calls Performance is linear? Exponential?

64. 64 Data Presentation The final results will be plotted as a function of the block size n