Ch1. Fundamentals of Computer Design 2. Performance ECE562/468 Advanced Computer Architecture Prof. Honggang Wang ECE Department University of Massachusetts Dartmouth 285 Old Westport Rd. North Dartmouth, MA Slides based on the PowerPoint Presentations created by David Patterson as part of the Instructor Resources for the textbook by Hennessy & Patterson Updated by Honggang Wang.

Administrative Issues (02/012/2015) Project team set-up due Thursday, Feb. 12 – Your group leader must send me an about your group information by 02/12 If you missed some classes, go to the course website for syllabus and 1 st lecture. 2

Review of Lecture #1 In the first lecture, we covered the Technology Trends 1.Define and quantity power 2.Define and quantity relative cost 3.Define and quantity dependability 3

Outline Careful, quantitative comparisons: Performance 1.Compute Mean 2.Benchmark 4

Definition: Performance 5 Performance is in events per sec: –bigger is better If we are primarily concerned with response time: the time between the start and the completion of an event (execution time). " X is n times faster than Y" means:

Performance: What to measure To increase predictability, collections of benchmark applications, called benchmark suites, are popular reliable measure of the performance is the execution time of real program the users would simply compare the execution time of their workloads a popular measure of performance of processors with a variety of applications SPECCPU: popular desktop benchmark suite – CPU only, split between integer and floating point programs – SPECint2000 has 12 integer, SPECfp2000 has 14 integer pgms – SPECCPU2006 to be announced Spring 2006 – SPECSFS (NFS file server) and SPECWeb (WebServer) added as server benchmarks Transaction Processing Council measures server performance and cost- performance for databases ( – TPC-C Complex query for Online Transaction Processing – TPC-H models ad hoc decision support – TPC-W a transactional web benchmark – TPC-App application server and web services benchmark 6

How Summarize Suite Performance (1/5) Arithmetic average of execution time of all pgms? – But they vary by 4X in speed, so some would be more important than others in arithmetic average Could add a weights per program, but how pick up weight? – Different companies want different weights for their products SPECRatio: Normalize execution times to reference computer, yielding a ratio proportional to performance 7

How Summarize Suite Performance (2/5) If program SPECRatio on Computer A is 1.25 times bigger than Computer B, then 8 Note that when comparing 2 computers as a ratio, execution times on the reference computer drop out, so choice of reference computer is irrelevant

How Summarize Suite Performance (3/5) Since ratios, proper mean is geometric mean (SPECRatio unitless, so arithmetic mean meaningless) 9 1. The SPECRatio are multiplied and then the nth root (where n is the count of SPECRatio in the set) of the resulting product is takennth rootproduct 2. indicates the central tendency or typical value of a set of numbers

How Summarize Suite Performance (3/5) Suppose an orange tree yields 100 oranges one year and then 180, 210 and 300 the following years, so the growth is 80%, 16.7% and 42.9% for each year respectively. Using the arithmetic mean calculates a (linear) average growth of 46.5% (80% % % divided by 3).arithmetic mean However, if we start with 100 oranges and let it grow 46.5% each year, the result is 314 oranges, not 300, so the linear average over- states the year-on-year growth. Instead, we can use the geometric mean. Growing with 80% corresponds to multiplying with 1.80, so we take the geometric mean of 1.80, and 1.429, i.e., thus the "average" growth per year is 44.3%. If we start with 100 oranges and let the number grow with 44.3% each year, the result is 300 oranges. 10

How Summarize Suite Performance (4/5) Does a single mean well summarize performance of programs in benchmark suite? Can decide if mean is a good predictor by characterizing variability of distribution using standard deviation Like geometric mean, geometric standard deviation is multiplicative rather than arithmetic Can simply take the logarithm of SPECRatios, compute the standard mean and standard deviation, and then take the exponent to convert back: 11

How Summarize Suite Performance (5/5) Standard deviation is more informative if know distribution has a standard form – bell-shaped normal distribution, whose data are symmetric around mean – lognormal distribution, where logarithms of data--not data itself--are normally distributed (symmetric) on a logarithmic scale For a lognormal distribution, we expect that 68% of samples fall in range 95% of samples fall in range Note: Excel provides functions EXP(), LN(), and STDEV() that make calculating geometric mean and multiplicative standard deviation easy 12

Example Standard Deviation (1/2) GM and multiplicative StDev of SPECfp2000 for Itanium 2 13 Outside 1 StDev

Example Standard Deviation (2/2) GM and multiplicative StDev of SPECfp2000 for AMD Athlon 14 Outside 1 StDev

Comments on Itanium 2 and Athlon Standard deviation of 1.98 for Itanium 2 is much higher-- vs so results will differ more widely from the mean, and therefore are likely less predictable Falling within one standard deviation: – 10 of 14 benchmarks (71%) for Itanium 2 – 11 of 14 benchmarks (78%) for Athlon Thus, the results are quite compatible with a lognormal distribution (expect 68%) 15

Fallacies and Pitfalls (1/2) Fallacies - commonly held misconceptions – When discussing a fallacy, we try to give a counter example. Pitfalls - easily made mistakes. – Often generalizations of principles true in limited context – Show Fallacies and Pitfalls to help you avoid these errors Fallacy: Benchmarks remain valid indefinitely – Once a benchmark becomes popular, tremendous pressure to improve performance by targeted optimizations or by aggressive interpretation of the rules for running the benchmark: “benchmarksmanship.” – 70 benchmarks from the 5 SPEC releases. 70% were dropped from the next release since no longer useful Pitfall: A single point of failure – Rule of thumb for fault tolerant systems: make sure that every component was redundant so that no single component failure could bring down the whole system (e.g, power supply) 16

Fallacies and Pitfalls (2/2) Fallacy: Rated MTTF of disks is 1,200,000 hours or  140 years, so disks practically never fail – manufactures will put thousands of disks in a room, run them for a few months and count the number that fail – But disk lifetime is 5 years  replace a disk every 5 years; on average, 28 replacements wouldn't fail – A better unit: % that fail (1.2M MTTF = 833 FIT) – Fail over lifetime: if had 1000 disks for 5 years = 1000*(5*365*24)*833 /10 9 = 36,485,000 / 10 6 = 37 = 3.7% (37/1000) fail over 5 yr hr MTTF) But this is under pristine conditions lifetime (1.2M – little vibration, narrow temperature range  no power failures Real world: 3% to 6% of SCSI drives fail per year – FIT or 150, ,000 hour MTTF [Gray & van Ingen 05] 17

Summary Quantify and summarize performance – Ratios, Geometric Mean, Multiplicative Standard Deviation 18

19 Things To Do Find your partners for the class project – Feb. 12, Thursday ( me the team information) Check out the class website about – lecture notes – reading assignments Next Topics Fundamentals of Computer Design 3. Principles