1. A Question to Ponder On [from last lecture]
In the CISC vs. RISC debate a key argument of the RISC movement was that because of its simplicity, RISC would always remain ahead.
If there were enough transistors to implement a CISC on chip, then those same transistors could implement a pipelined RISC
If there was enough to allow for a pipelined CISC there would be enough to have an on-chip cache for RISC. And so on.
After 20 years of this debate what do you think?
Hint: Think of commercial PC’s, Moore’s law and some of the data in the first chapter of the book (and on these slides)

2. CIS 775:Lecture 2
Performance and Cost

3. The Bottom Line: Performance (and Cost)
Plane
DC to Paris
6.5 hours
3 hours
Speed
610 mph
1350 mph
Passengers
470
132
Throughput (pmph)
286,700
178,200
Boeing 747
BAD/Sud Concodre
Fastest for 1 person?
Which takes less time to transport 470 passengers?
Time to run the task (ExTime)
Execution time, response time, latency
Tasks per day, hour, week, sec, ns … (Performance)
Throughput, bandwidth

4. The Bottom Line: Performance (and Cost)
"X is n times faster than Y" means
ExTime(Y) Performance(X) =
ExTime(X) Performance(Y)
Speed of Concorde vs. Boeing 747
Throughput of Boeing 747 vs. Concorde
1350 / 610 = 2.2X
286,700/ 178, X

6. Amdahl’s Law
ExTimenew = ExTimeold x (1 - Fractionenhanced) + Fractionenhanced
Speedupenhanced 1
ExTimeold
ExTimenew
Speedupoverall = =
(1 - Fractionenhanced) + Fractionenhanced
Speedupenhanced

7. Amdahl’s Law
Floating point instructions improved to run 2X; but only 10% of actual instructions are FP
ExTimenew =
Speedupoverall =

8. Aspects of CPU Performance
CPU time = Seconds = Instructions x Cycles x Seconds
Program Program Instruction Cycle
Inst Count CPI Clock Rate
Program X
Compiler X (X)
Inst. Set X X
Organization X X
Technology X

9. Cycles Per Instruction
“Average Cycles per Instruction”
CPI = (CPU Time * Clock Rate) / Instruction Count
= Cycles / Instruction Count n
CPU time = CycleTime *  CPI * I i i
i = 1
“Instruction Frequency” n
CPI =  CPI * F where F = I i i i i
i = 1
Instruction Count
Invest Resources where time is Spent!

10. Example: Calculating CPI
Base Machine (Reg / Reg)
Op Freq Cycles CPI(i) (% Time)
ALU 50% (33%)
Load 20% (27%)
Store 10% (13%)
Branch 20% (27%)
1.5
Typical Mix

12. Metrics of Performance
Application
Answers per month
Operations per second
Programming
Language
Compiler
(millions) of Instructions per second: MIPS
(millions) of (FP) operations per second: MFLOP/s
ISA
Datapath
Megabytes per second
Control
Function Units
Cycles per second (clock rate)
Transistors
Wires
Pins

13. Measurement and Evaluation
Architecture is an iterative process:
Searching the space of possible designs
At all levels of computer systems
Creativity
Cost /
Performance
Analysis
Good Ideas
Mediocre Ideas
Bad Ideas

14. Computer Engineering Methodology
Evaluate Existing
Systems for
Bottlenecks
Implementation
Complexity
Benchmarks
Technology
Trends
How hard to build
Importance of simplicity (wearing a seat belt); avoiding a personal disaster
Theory vs. practice
Implement Next
Generation System
Simulate New
Designs and
Organizations
Workloads

15. Measurement Tools Benchmarks, Traces, Mixes
Hardware: Cost, delay, area, power estimation
Simulation (many levels)
ISA, RT, Gate, Circuit
Queuing Theory
Rules of Thumb
Fundamental “Laws”/Principles
Understanding the limitations of any measurement tool is crucial.

16. SPEC First Round One program: 99% of time in single line of code
New front-end compiler could improve dramatically

17. Cases of Benchmark Engineering
The motivation is to tune the system to the benchmark to achieve peak performance.
At the architecture level
Specialized instructions
At the compiler level (compiler flags)
Blocking in Spec89  factor of 9 speedup
Incorrect compiler optimizations/reordering.
Would work fine on benchmark but not on other programs
I/O level
Spec92 spreadsheet program (sp)
Companies noticed that the produced output was always out put to a file (so they stored the results in a memory buffer) and then expunged at the end (which was not measured).
One company eliminated the I/O all together.

18. After putting in a blazing performance on the benchmark test,
Sun issued a glowing press release claiming that it had
outperformed Windows NT systems on the test.
Pendragon president Ivan Phillips cried foul, saying the results
weren't representative of real-world Java performance and that
Sun had gone so far as to duplicate the test's code within Sun's
Just-In-Time compiler. That's cheating, says Phillips, who claims
that benchmark tests and real-world applications aren't
the same thing.
Did Sun issue a denial or a mea culpa? Initially, Sun neither
denied optimizing for the benchmark test nor apologized for
it. "If the test results are not representative of real-world Java
applications, then that's a problem with the benchmark,"
Sun's Brian Croll said.
After taking a beating in the press, though, Sun retreated and
issued an apology for the optimization.[Excerpted from PC Online 1997]

19. Issues with Benchmark Engineering
Motivated by the bottom dollar, good performance on classic suites  more customers, better sales.
Benchmark Engineering  Limits the longevity of benchmark suites
Technology and Applications Limits the longevity of benchmark suites.

20. SPEC: System Performance Evaluation Cooperative
First Round 1989
10 programs yielding a single number (“SPECmarks”)
Second Round 1992
SPECInt92 (6 integer programs) and SPECfp92 (14 floating point programs)
Compiler Flags unlimited. March 93
new set of programs: SPECint95 (8 integer programs) and SPECfp95 (10 floating point)
“benchmarks useful for 3 years”
Single flag setting for all programs: SPECint_base95, SPECfp_base95

21. Reporting Performance Results
Reproducability
 Apply them on publicly available benchmarks. Pecking/Picking order
Real Programs
Real Kernels
Toy Benchmarks
Synthetic Benchmarks

22. How to Summarize Performance
Arithmetic mean (weighted arithmetic mean) tracks execution time: sum(Ti)/n or sum(Wi*Ti)
Harmonic mean (weighted harmonic mean) of rates (e.g., MFLOPS) tracks execution time: n/sum(1/Ri) or n/sum(Wi/Ri)
Normalized execution time is handy for scaling performance (e.g., X times faster than SPARCstation 10)
But do not take the arithmetic mean of normalized execution time, use the geometric mean = (Product(Ri)^1/n)

23. Performance Evaluation
“For better or worse, benchmarks shape a field”
Good products created when have:
Good benchmarks
Good ways to summarize performance
Given sales is a function in part of performance relative to competition, investment in improving product as reported by performance summary
If benchmarks/summary inadequate, then choose between improving product for real programs vs. improving product to get more sales; Sales almost always wins!
Execution time is the measure of computer performance!

24. Simulations When are simulations useful?
What are its limitations, I.e. what real world phenomenon does it not account for?
The larger the simulation trace, the less tractable the post-processing analysis.

25. Queueing Theory
What are the distributions of arrival rates and values for other parameters?
Are they realistic?
What happens when the parameters or distributions are changed?

26. Chapter Summary, #1 Designing to Last through Trends
Capacity Speed
Logic 2x in 3 years 2x in 3 years
DRAM 4x in 3 years 2x in 10 years
Disk 4x in 3 years 2x in 10 years
6yrs to graduate => 16X CPU speed, DRAM/Disk size
Time to run the task
Execution time, response time, latency
Tasks per day, hour, week, sec, ns, …
Throughput, bandwidth
“X is n times faster than Y” means
ExTime(Y) Performance(X) =
ExTime(X) Performance(Y)

27. Chapter Summary, #2 Amdahl’s Law: CPI Law:
Execution time is the REAL measure of computer performance!
Good products created when have:
Good benchmarks, good ways to summarize performance
Die Cost goes roughly with die area4
Speedupoverall =
ExTimeold
ExTimenew = 1
(1 - Fractionenhanced) + Fractionenhanced
Speedupenhanced
CPU time = Seconds = Instructions x Cycles x Seconds
Program Program Instruction Cycle

28. Food for thought
Two companies reports results on two benchmarks one on a Fortran benchmark suite and the other on a C++ benchmark suite.
Company A’s product outperforms Company B’s on the Fortran suite, the reverse holds true for the C++ suite. Assume the performance differences are similar in both cases.
Do you have enough information to compare the two products. What information will you need?

29. Amdahl’s Law (answer)
Floating point instructions improved to run 2X; but only 10% of actual instructions are FP
ExTimenew = ExTimeold x ( /2) = 0.95 x ExTimeold 1
Speedupoverall = =
1.053
0.95