CS 360 Lecture 11

 In most computer systems:  The cost of people (development) is much greater than the cost of hardware  Yet, performance is important  A single bottleneck can slow down an entire system  Future loads may be much greater than predicted 2

 Real time systems  when computation must be fast enough to support the service provided, e.g., fly- by-wire control systems have tight response time requirements.  Very large computations  where elapsed time may be measured in days, e.g., calculation of weather forecasts must be fast enough for the forecasts to be useful.  User interfaces  where humans have high expectations, e.g., mouse tracking must appear instantaneous.  Transaction processing  where staff need to be productive and customers not annoyed by delays, e.g., airline check-in. 3

 High-performance computing:  Large data collections (e.g., Amazon)  Huge numbers of users (e.g., Google)  Large computations (e.g., weather forecasting)  Must balance cost of hardware against cost of so(ware development  Some configurations are very difficult to program and debug  Sometimes it is possible to isolate applications programmers from the system complexities 4

 Tasks  Predict performance problems before a system is created.  Design and build a system that is not vulnerable to performance problems.  Identify causes and fix problems after a system is implemented. 5

 Examples  In a distributed system, what messages pass between nodes?  How many times must the system read from disk for a single transaction?  What buffering and caching is used?  Are operations in parallel or sequential?  Are other systems competing for a shared resource (e.g., a network or server farm)?  How does the operating system schedule tasks? 6

 Usually, CPU performance is not the limiting factor.  Hardware bottlenecks  Reading data from disk  Shortage of memory (including paging)  Moving data from memory to CPU  Network capacity (bandwidth)  Inefficient software  Algorithms that do not scale well  Parallel and sequential processing 7

 CPU performance is a limi1ng constraint in certain domains:  large data analysis (e.g., searching)  mathematical computation (e.g., engineering)  compression and encryption  multimedia (e.g., video)  perception (e.g., image processing) 8

9 Actual performance may be considerably less than theoretical peak performance

 Utilization is the proportion of the capacity of a service that is used on average.  Utilization =  avg. service time for a transaction/avg. arrival time of transactions  Peak loads and temporary increases in demand can be much greater than the average. 10

 Direct measurement on subsystem (benchmark)  Disk I/O  CPU  Network  Simulation  All require detailed understanding of the interaction between software and hardware systems. 11

 Tools for examining performance of subsystems:  Ex: Sysbench  Sysbench is a modular, cross-platform, multi-threaded benchmark tool used to quickly gain information about system performance.  Tests the following:  File I/O performance  Job scheduler performance  Memory allocation and transfer speed  Database service performance 12

 Build a computer program that models the system as set of states and events advance simulated time determine which events occurred update state and event list repeat  Discrete time simulation:  Time is advanced in fixed steps (e.g., 1 ms)  Next event simulation:  Time is advanced to next event 13

 Benchmarks:  Run tools on standard problem sets, sample inputs, or simulated loads on the system  Collect data on resource availability, utilization, performance, etc.  Example:  Disk I/O benchmark using MySQL to determine performance of different cloud stacks (OpenStack, Eucalyptus, Docker, etc.)  Network, base OS, hardware should remain the same across all tests. 14

15 Benchmark: Throughput vs. number of CPU cores utilized

 If a system performs badly, begin by identifying the cause:  Instrumentation:  Add timers to the code. Often this will reveal that delays are centered in a specific part of the system.  Test loads:  Run the system with varying loads (high transaction rates, large input files, many users, etc.). This may reveal the characteristics of when the system runs badly.  Design and code reviews:  Team review of system design, program design, and suspect sections of code. This may reveal an algorithm that is running very slowly, e.g., a sort, locking procedure, etc.  Find the underlying cause and fix it or the problem will return! 16

 Original version:  The density of transistors in an integrated circuit will double every year.  Gordon Moore, Intel, 1965  Current version:  Cost/performance of silicon chips doubles every 18 months. 17

 Silicon chips: cost/performance improves 30% / year  in 12 years = 20x  in 24 years = 500x  Storage media: cost/performance improves 40% / year  in 12 years = 50x  in 24 years = 3,000x  These assumptions are conservative. During some periods, the increases have been considerably faster.  Note: Recently, the rate of performance increase in individual components, such as CPUs, has slowed down, but the overall rate of increase has been maintained by placing many CPU cores on a single chip. 18

 Will this be a typical laptop?  Surely there will be some fundamental changes in how these resources are packaged and used. 19

 Original:  Work expands to fill the time available.  C. Northcote Parkinson  Software development version:  Demand will expand to use all the hardware available.  Low prices will create new demands.  Your software will be used on equipment that you have not envisioned. 20

 Be careful about the assumptions that you make Here are some past assump1ons that caused problems:  Unix file system will never exceed 2 GB.  AppleTalk networks will never have more than 256 hosts (28 bits).  GPS software will not last more than 1024 weeks.  Two bits are sufficient to represent a year (Y2K bug).  Etc., etc.,

 Performance specifications should, at minimum, cover the following concepts:  In detail, what is the performance test scope?  What subsystems, interfaces, components, etc. are in and out of scope of the specific performance test?  For the user interfaces, how many concurrent users are expected for each?  Specify peak vs. normal usage  What does the target system (hardware) look like?  Specify all hardware configurations 22

 Performance specifications should, at minimum, cover the following concepts:  What is the Application Workload Mix for each component?  Ex: 20% login, 40% search, 30% item select, 10% checkout  What is the System Workload Mix?  What processes are running?  What are the time requirements for any/all processes?  Specify peak vs. normal 23

 The performance requirements should include test cases to demonstrate performance of  Individual components,  Interfaces,  Groups of components,  And the system as whole 24

Tasks needed for performance documentation: 1. Configure the testing environment 2. Initial test run – to check correctness of testing script 3. Execute tests (repeatedly) 4. Record and analyze the results (Pass/Fail)  Investigate corrective action if test fails 5. Re-tune components/interfaces/system and repeat process 25

Tasks needed for performance documentation: 6. Gathering performance requirements 7. Decide whether to use internal or external resources to perform tests. 8. Developing a plan to test for performance 9. Specify testing tools 10. Specify test data 11. Develop proof of concept test scripts for each component/interface 26