Using Standard Industry Benchmarks Chapter 7 CSE807

Reasons For Using Benchmarks Real Workload impractical to set up Costly, hard to measure Unknown for most part Benchmarks Set of well-defined representative programs to be run on different systems and networks to compare performance

Nature of Benchmarks Measurable and Repeatable Used for Monitoring and Diagnostics Used for Capacity Planning Can be confusing if customer workload does not match with the benchmark Best for comparing systems Complementary source of information in capacity studies

Nature of Benchmarks (cont.) Users want fast Response Time and high availability Managers ask –How many transactions/minute can the system handle –How many requests/minute can the web site service –What is the system’s operational cost –What is the initial hardware & installation cost –How to determine standard measure of system performance

Performance Measures Server Measures –CPU speed – MIPS - millions of instructions per second RISC - Reduced Instruction Set Computer CISC - Complex Instructions Set Computer Web Servers Measures –Inclusive of Clients and Server –Network related effects –Throughput –Latency

Benchmarks Hierarchy Synthetic - Basic Operations, very limited utility –Dhrystone -Speed for fixed point computations –Whetstone -Speed for floating point computations Toy Benchmarks –Small programs implementing classical puzzles –Does not help in predicting performance for any real workload

Benchmarks Hierarchy (cont.) Kernels – Livermore Loops and Linpack – Mainly for CPU performance – Not used for performance perceived by users Real Programs – SPEC and TPC – C compiler, UNIX utilities, debit & credit bank transactions etc. – Used to obtain most accurate picture of the system performance perceived by the user

Avoiding Pitfalls Understand the benchmark environment –Processor specification, memory, I/O subsystem, network and software(OS, DB) Compare your system with benchmark –Similarity and dissimilarity in the environment Representativeness of workload –Similarity and dissimilarity of the workload

Properties of a good benchmark Relevant - it must provide meaningful performance measure within a specific problem domain Understandable - results should be simple and easy to understand Scaleable - must be applicable to wide range of systems (costs, performance) Acceptable - should present unbiased results that are recognized by users and vendors

Component Level Benchmarks CPU - SPECxx –CINT Compute-intensive Integer performance –CFP Floating point performance –Designed for performance of computer processor, memory architecture and compiler –SPECratio is Ratio of Reference time to run time –SPECint is the geometric mean of eight normalized ratios

Component Level (cont.) File Servers - synthetic benchmark that model a workload of input mix to file server –LADDIS - measure NFS server performance –measured at various load levels –generates throughput and average response time –parameters can be modified and adjusted to get a workload representing a user environment –50 msec is an arbitrary reference point

System Level Benchmarks Transaction Processing Systems Measure the CPU, I/O subsystem, the network, database, compilers and the OS TPC - Transaction Processing Council –TPC-B measures database transactions –TPC-A also measures network performance –TPC-C measure order-entry applications –TPC-D for decision support systems

System Level Benchmarks(cont.) Web Servers - simulate web browsers (assume no transmission errors) Webstone - simulates server and client processes, spawns a predefined number of clients for HTTP requests to server –Results include Throughput and Latency –Little’s Load Factor (LLF) gives degree of concurrency –Avg. # of connections equal to Connection rate times Avg. residence time

System Level Benchmarks(cont.) SPECweb - uses logs from popular web servers –workload parameters are fixed unlike LADDIS –result of SPECweb is the server’s maximum throughput –Response time is for server only, does not include network delays