Software System Performance CS 560

Performance of computer systems In most computer systems:  The cost of people (development) is much greater than the cost of hardware Yet, performance is important  A single bottleneck can slow down an entire system  Loads/resource usage may be much greater than predicted 2

High-performance Computing High-performance computing:  Large data collections (e.g., Amazon)  How to transfer/modify/view this data?  How to store this data?  Huge numbers of users (e.g., Google)  How to allow many thousands of concurrent users?  Synchronizing of user data across many devices  Large computations (e.g., weather forecasting)  Computing models  Processing and storage of complex information 3

Performance challenges for all software systems Tasks  Predict performance problems before a system is created.  Design and build a system that is not vulnerable to performance problems.  Is this possible?  Identify causes and fix problems after a system is implemented. 4

Interactions between hardware and software Examples  In a distributed system, what messages pass between nodes?  How many times must the system read from disk for a single transaction?  What buffering and caching is used?  Are operations in parallel or sequential?  Are other systems competing for a shared resource (e.g., a network or storage device)?  How does the operating system schedule tasks? 5

Look for bottlenecks Usually, CPU performance is not the limiting factor. Hardware bottlenecks  Reading data from disk  Shortage of memory (including paging)  Moving data from memory to CPU  Network capacity (bandwidth) Inefficient software  Algorithms that do not scale well  Parallel and sequential processing 6

Look for bottlenecks CPU performance is a limiting constraint in certain domains:  large data analysis (e.g., searching)  mathematical computation (e.g., engineering)  compression and encryption  multimedia (e.g., video)  perception (e.g., image processing) 7

Timescale of different components 8 Actual performance may be considerably less than theoretical peak performance

Look for bottlenecks: utilization Utilization is the proportion of the capacity of a service that is used on average.  Utilization =  avg. service time for a transaction/avg. arrival time of transactions Peak loads and temporary increases in demand can be much greater than the average.  How to accommodate peak and temporary increases?  Tradeoff for performance? 9

Predicting system performance Direct measurement on subsystem (benchmark)  Disk I/O  CPU  Network Simulation  Based on component anticipated performance  Used generally when physical resources aren't available Both require detailed understanding of the interaction between software and hardware systems. 10

Direct measurement on subsystem Tools for examining performance of subsystems:  Ex: Sysbench  Sysbench is a modular, cross-platform, multi-threaded benchmark tool used to quickly gain information about system performance.  Tests the following:  File I/O performance  Job scheduler performance  Memory allocation and transfer speed  Database service performance 11

Simulations Build a computer program that models the system as set of states and events advance simulated time determine which events occurred update state and event list repeat Discrete time simulation:  Time is advanced in fixed steps (e.g., 1 ms) Next event simulation:  Time is advanced to next event 12

Measurements on OS/architectures Benchmarks:  Run tools on standard problem sets, sample inputs, or simulated loads on the system  Collect data on resource availability, utilization, performance, etc. Example :  Disk I/O benchmark using MySQL to determine performance of different cloud stacks (OpenStack, Eucalyptus, Docker, etc.)  Network, base OS, hardware should remain the same across all tests. 13

Fixing bad performance If a system performs badly, begin by identifying the cause: Instrumentation:  Add timers to the code. Often this will reveal that delays are centered in a specific part of the system. Test loads:  Run the system with varying loads (high transaction rates, large input files, many users, etc.). This may reveal the characteristics of when the system runs badly. Design and code reviews:  Team review of system design, program design, and suspect sections of code. This may reveal an algorithm that is running very slowly, e.g., a sort, locking procedure, etc. Find the underlying cause and fix it or the problem will return! 14

Predicting performance change: Moore’s Law Original version:  The density of transistors in an integrated circuit will double every year.  Gordon Moore, Intel, 1965 Current version:  Cost/performance of silicon chips doubles every 18 months.  Moving closer to 24 months as Intel’s tick-tock cycle becomes exceedingly difficult. 15

Moore’s law: rules of thumb Silicon chips: cost/performance improves 30% / year  in 12 years = 20x  in 24 years = 500x Storage media: cost/performance improves 40% / year  in 12 years = 50x  in 24 years = 3,000x These assumptions are conservative. During some periods, the increases have been considerably faster. 16

Moore’s law example Will this be a typical laptop? Surely there will be some fundamental changes in how these resources are packaged and used. 17

Parkinson’s law Original (1955):  Work expands to fill the time available.  C. Northcote Parkinson Software development version:  Demand will expand to use all the hardware available.  Low prices will create new demands.  Your software will be used on equipment that you have not envisioned. 18

False assumptions from the past Be careful about the assumptions that you make:  Unix file system will never exceed 2 GB.  AppleTalk networks will never have more than 256 hosts (1 Byte address space).  GPS software will not last more than 1024 weeks.  Two bits are sufficient to represent a year (Y2K bug).  Etc., etc.,

Software Performance Requirements (documentation) Performance specifications should, at minimum, cover the following concepts:  In detail, what is the performance test scope?  What subsystems, interfaces, components, etc. are in and out of scope of the specific performance test?  For the user interfaces, how many concurrent users are expected?  Specify peak vs. normal usage  What does the target system (hardware) look like?  Specify all hardware configurations 20

Software Performance Requirements (documentation) Performance specifications should, at minimum, cover the following concepts:  What is the Application Workload Mix for each component?  Ex: 20% login, 40% search, 30% item select, 10% checkout  What is the System Workload Mix?  What processes are running?  What are the time requirements for any/all processes?  Specify peak vs. normal 21

Software Performance Requirements (documentation) The performance requirements should include test cases to demonstrate performance of  Individual components  Interfaces  Groups of components  The entire system at once 22

Software Performance Requirements (documentation) Tasks needed for performance documentation: 1.Gather performance requirements 2.Developing a plan to test for performance 3.Specify testing tools 4.Specify test data to be used 5.Decide whether to use internal or external resources to perform tests. 6.Develop proof of concept test scripts for each component/interface 23

Software Performance Requirements (documentation) Tasks needed for performance documentation: 7.Configure the testing environment 8.Initial test run – to check correctness of testing script 9.Execute tests (repeatedly) 10.Record and analyze the results, pass/fail based on performance requirements  Investigate corrective action if test fails 11.Re-tune components/interfaces/system and repeat process 24