Lecture 2c: Benchmarks. Benchmarking Benchmark is a program that is run on a computer to measure its performance and compare it with other machines Best.

Slides:



Advertisements
Similar presentations
11 Measuring performance Kosarev Nikolay MIPT Feb, 2010.
Advertisements

CS2100 Computer Organisation Performance (AY2014/2015) Semester 2.
ECE 4100/6100 Advanced Computer Architecture Lecture 3 Performance Prof. Hsien-Hsin Sean Lee School of Electrical and Computer Engineering Georgia Institute.
Performance Analysis of Multiprocessor Architectures
Chapter 4 M. Keshtgary Spring 91 Type of Workloads.
2-1 ECE 361 ECE C61 Computer Architecture Lecture 2 – performance Prof. Alok N. Choudhary
Chapter 1 CSF 2009 Computer Performance. Defining Performance Which airplane has the best performance? Chapter 1 — Computer Abstractions and Technology.
CSCE 212 Chapter 4: Assessing and Understanding Performance Instructor: Jason D. Bakos.
Chapter 4 Assessing and Understanding Performance Bo Cheng.
CIS629 Fall Lecture Performance Overview Execution time is the best measure of performance: simple, intuitive, straightforward. Two important.
Computer Performance Evaluation: Cycles Per Instruction (CPI)
Computer ArchitectureFall 2007 © September 17, 2007 Karem Sakallah CS-447– Computer Architecture.
CS/ECE 3330 Computer Architecture Chapter 1 Performance / Power.
1 Lecture 11: Digital Design Today’s topics:  Evaluating a system  Intro to boolean functions.
Chapter 4 Assessing and Understanding Performance
1 Lecture 10: FP, Performance Metrics Today’s topics:  IEEE 754 representations  FP arithmetic  Evaluating a system Reminder: assignment 4 due in a.
CIS429/529 Winter 07 - Performance - 1 Performance Overview Execution time is the best measure of performance: simple, intuitive, straightforward. Two.
Rung-Bin Lin Chapter 1: Fundamental of Computer Design1-1 Chapter 1. Fundamentals of Computer Design Introduction –Performance Improvement due to (1).
Using Standard Industry Benchmarks Chapter 7 CSE807.
Chapter 1 Section 1.4 Dr. Iyad F. Jafar Evaluating Performance.
Lecture 2: Technology Trends and Performance Evaluation Performance definition, benchmark, summarizing performance, Amdahl’s law, and CPI.
Performance & Benchmarking. What Matters? Which airplane has best performance:
1 Computer Performance: Metrics, Measurement, & Evaluation.
Where Has This Performance Improvement Come From? Technology –More transistors per chip –Faster logic Machine Organization/Implementation –Deeper pipelines.
Chapter 1 Computer Abstractions and Technology Part II.
Lecture 2b: Performance Metrics. Performance Metrics Measurable characteristics of a computer system: Count of an event Duration of a time interval Size.
BİL 221 Bilgisayar Yapısı Lab. – 1: Benchmarking.
Recap Technology trends Cost/performance Measuring and Reporting Performance What does it mean to say “computer X is faster than computer Y”? E.g. Machine.
Lecture 2: Performance Measurement
1 Introduction Background: CS 3810 or equivalent, based on Hennessy and Patterson’s Computer Organization and Design Text for CS/EE 5810/6810: Hennessy.
C OMPUTER O RGANIZATION AND D ESIGN The Hardware/Software Interface 5 th Edition Chapter 1 Computer Abstractions and Technology Sections 1.5 – 1.11.
CDA 3101 Fall 2013 Introduction to Computer Organization Computer Performance 28 August 2013.
Lecture 9: 9/24/2002CS170 Fall CS170 Computer Organization and Architecture I Ayman Abdel-Hamid Department of Computer Science Old Dominion University.
1. 2 Table 4.1 Key characteristics of six passenger aircraft: all figures are approximate; some relate to a specific model/configuration of the aircraft.
From lecture slides for Computer Organization and Architecture: Designing for Performance, Eighth Edition, Prentice Hall, 2010 CS 211: Computer Architecture.
1 CS/COE0447 Computer Organization & Assembly Language CHAPTER 4 Assessing and Understanding Performance.
Computer Architecture
1 Seoul National University Performance. 2 Performance Example Seoul National University Sonata Boeing 727 Speed 100 km/h 1000km/h Seoul to Pusan 10 hours.
Performance Lecture notes from MKP, H. H. Lee and S. Yalamanchili.
CEN 316 Computer Organization and Design Assessing and Understanding Performance Mansour AL Zuair.
Cost and Performance.
Linear Algebra Libraries: BLAS, LAPACK, ScaLAPACK, PLASMA, MAGMA
CSE2021: Computer Organization Instructor: Dr. Amir Asif Department of Computer Science York University Handout # 2: Measuring Performance Topics: 1. Performance:
Performance Performance
1 Lecture 2: Performance, MIPS ISA Today’s topics:  Performance equations  MIPS instructions Reminder: canvas and class webpage:
September 10 Performance Read 3.1 through 3.4 for Wednesday Only 3 classes before 1 st Exam!
3/12/2013Computer Engg, IIT(BHU)1 PARALLEL COMPUTERS- 3.
Performance Analysis Topics Measuring performance of systems Reasoning about performance Amdahl’s law Systems I.
CMSC 611: Advanced Computer Architecture Performance & Benchmarks Some material adapted from Mohamed Younis, UMBC CMSC 611 Spr 2003 course slides Some.
Jan. 5, 2000Systems Architecture II1 Machine Organization (CS 570) Lecture 2: Performance Evaluation and Benchmarking * Jeremy R. Johnson Wed. Oct. 4,
CS203 – Advanced Computer Architecture Performance Evaluation.
BITS Pilani, Pilani Campus Today’s Agenda Role of Performance.
Chapter 1 Performance & Technology Trends. Outline What is computer architecture? Performance What is performance: latency (response time), throughput.
June 20, 2001Systems Architecture II1 Systems Architecture II (CS ) Lecture 1: Performance Evaluation and Benchmarking * Jeremy R. Johnson Wed.
Measuring Performance II and Logic Design
CS203 – Advanced Computer Architecture
Lecture 2: Performance Evaluation
CS161 – Design and Architecture of Computer Systems
4- Performance Analysis of Parallel Programs
September 2 Performance Read 3.1 through 3.4 for Tuesday
ECE 4100/6100 Advanced Computer Architecture Lecture 1 Performance
Morgan Kaufmann Publishers
CSCE 212 Chapter 4: Assessing and Understanding Performance
CMSC 611: Advanced Computer Architecture
Performance of computer systems
Performance of computer systems
CS170 Computer Organization and Architecture I
CMSC 611: Advanced Computer Architecture
Performance of computer systems
COMS 361 Computer Organization
Presentation transcript:

Lecture 2c: Benchmarks

Benchmarking Benchmark is a program that is run on a computer to measure its performance and compare it with other machines Best benchmark is the users’ workload – the mixture of programs and operating system commands that users run on a machine.  Not practical Standard benchmarks

Benchmarking Types of Benchmarks Synthetic benchmarks Toy benchmarks Microbenchmarks Program Kernels Real Applications

Benchmarking Synthetic benchmarks Artificially created benchmark programs that represent the average frequency of operations (instruction mix) of a large set of programs Whetstone benchmark Dhrystone benchmark Rhealstone benchmark

Benchmarking Synthetic benchmarks Whetstone benchmark First written in Algol60 in 1972, today Fortran, C/C++, Java versions are available Represents the workload of numerical applications Measures floating point arithmetic performance Unit is Millions of Whetstone instructions per second (MWIPS) Shortcommings: Does not represent constructs in modern languages, such as pointers, etc. Does not consider cache effects

Benchmarking Synthetic benchmarks Dhrystone benchmark First written in Ada in1984, today Represents the workload of C version is available Statistics are collected on system software, such as operating system, compilers, editors and a few numerical programs Measures integer and string performance, no floating-point operations Unit is the number of program iteration completions per second Shortcommings: Does not represent real life programs Compiler optimization overstates system performance Small code that may fit in the instruction cache

Benchmarking Synthetic benchmarks Rhealstone benchmark Multi-tasking real-time systems Factors are: Task switching time Pre-emption time Interrupt latency time Semaphore shuffling time Dead-lock breaking time Datagram throughput time Metric is Rhealstones per second 6 ∑ w i. (1/ t i ) i=1

Benchmarking Toy benchmarks lines of code that the result is known before running the toy program Quick sort Sieve of Eratosthenes Finds prime numbers func sieve( var N ) var PrimeArray as array of size N initialize PrimeArray to all true for i from 2 to N for each j from i + 1 to N, where i divides j set PrimeArray( j ) = false

Benchmarking Microbenchmarks Small, specially designed programs used to test some specific function of a system (eg. Floating-point execution, I/O subsystem, processor-memory interface, etc.) Provide values for important parameters of a system Characterize the maximum performance if the overall performance is limited by that single component

Benchmarking Kernels Key pieces of codes from real applications. LINPACK and BLAS Livermore Loops NAS

Benchmarking Kernels LINPACK and BLAS Libraries LINPACK – linear algebra package Measures floating-point computing power Solves system of linear equations Ax=b with Gaussian elimination Metric is MFLOP/s DAXPY - most time consuming routine Used as the measure for TOP500 list BLAS – Basic linear algebra subprograms LINPACK makes use of BLAS library

Benchmarking Kernels LINPACK and BLAS Libraries SAXPY – Scalar Alpha X Plus Y Y =  X + Y, where X and Y are vectors,  is a scalar SAXPY for single and DAXPY for double precision Generic implementation: for (int i = m; i < n; i++) { y[i] = a * x[i] + y[i]; }

Benchmarking Kernels Livermore Loops Developed at LLNL Originally in Fortran, now also in C 24 numerical application kernels, such as: hydrodynamics fragment, incomplete Cholesky conjugate gradient, inner product, banded linear systems solution, tridiagonal linear systems solution, general linear recurrence equations, first sum, first difference, 2-D particle in a cell, 1-D particle in a cell, Monte Carlo search, location of a first array minimum, etc. Metrics are arithmetic, geometric and harmonic mean of CPU rate

Benchmarking Kernels NAS Parallel Benchmarks Developed at NASA Advanced Supercomputing division Paper-and-pencil benchmarks 11 benchmarks, such as: Discrete Poisson equation, Conjugate gradient Fast Fourier Transform Bucket sort Embarrassingly parallel Nonlinear PDE solution Data traffic, etc.

Benchmarking Real Applications Programs that are run by many users C compiler Text processing software Frequently used user applications Modified scripts used to measure particular aspects of system performance, such as interactive behavior, multiuser behavior

Benchmarking Benchmark Suites Desktop Benchmarks SPEC benchmark suite Server Benchmarks SPEC benchmark suite TPC Embedded Benchmarks EEMBC

Benchmarking SPEC Benchmark Suite Desktop Benchmarks CPU-intensive SPEC CPU integer (CINT2000) and 14 floating-point (CFP2000) benchmarks Real application programs: C compiler Finite element modeling Fluid dynamics, etc. Graphics intensive SPECviewperf Measures rendering performance using OpenGL SPECapc Pro/Engineer – 3D rendering with solid models Solid/Works – 3D CAD/CAM design tool, CPU-intensive and I/O intensive tests Unigraphics – solid modeling for an aircraft design Server Benchmarks SPECWeb – for web servers SPECSFS – for NFS performance, throughput-oriented

Benchmarking TPC Benchmark Suite Server Benchmark Transaction processing (TP) benchmarks Real applications TPC-C: simulates a complex query environment TPC-H: ad hoc decision support TPC-R: business decision support system where users run a standard set of queries TPC-W: business-oriented transactional web server Measures performance in transactions per second. Throughput performance is measured only when response time limit is met. Allows cost-performance comparisons

Benchmarking EEMBC Benchmarks for embedded computing systems 34 benchmarks from 5 different application classes: Automotive/industrial Consumer Networking Office automation Telecommunications

Benchmarking Benchmarking Strategies Fixed-computation benchmarks Fixed-time benchmarks Variable-computation and variable-time benchmarks

Benchmarking Benchmarking Strategies Fixed-computation benchmarks Fixed-time benchmarks Variable-computation and variable-time benchmarks

Benchmarking Fixed-Computation benchmarks W: fixed workload (number of instructions, number of floating-point operations, etc) T: measured execution time R: speed Compare

Benchmarking Fixed-Computation benchmarks Amdahl’s Law

Benchmarking Fixed-Time benchmarks On a faster system, a larger workload can be processed in the same amount of time T: fixed execution time W: workload R: speed Compare

Benchmarking Fixed-Time benchmarks Scaled Speedup

Benchmarking Variable-Computation and Variable-Time benchmarks In this type of benchmark, quality of the solution is improved. Q: quality of the solution T: execution time Quality improvements per second: