1. MisSPECulation Partial and Misleading use of SPEC CPU2000 in Computer Architecture Conferences Dave Kroondyk

2. Abstract 173 papers surveyed 115 used benchmarks from SPEC CINT 23 used entire suite Results are speculative Violation of Amdahl’s Law Unwarranted disregard for CFP2000 Higher data cache miss rate

3. SPEC CPU2000 2 suites CINT2000 for integer applications (12) CFP2000 for floating point applications (14)

4. ISCA 2002 27 papers published 23 used SPEC 4 of 16 used all of CINT 2 of 14 used all of CFP

5. CINT2000 benchmark usage Median = 7

6. Why?

7. Why? (cont.) Chosen benchmarks stress problem paper solves Violation of Amdahl’s Law Several benchmarks couldn’t be adapted to the environment used Understandable unless more than half can’t be run

8. Amdahl’s Law “The performance improvement to be gained from using some faster mode of execution is limited by the fraction of the time the faster mode can be used”

9. Applying Amdahl’s Law

10. CFP2000 Used less and less benchmarks are used per paper 36% of suite vs. 58% for CINT Why? FP applications are built around highly predictable and parallelizable loops Less branches are performed Higher prediction rate Instruction cache miss rate is lower

11. However… Data cache miss rate is much higher Suggests that memory-hierarchy oriented research should put emphasis on improving CFP2000 benchmarks But they don’t 35% use CFP2000 61% use CINT2000

12. Summary Misuse of tools leads to speculative and misleading results Applying Amdahl’s law “steals their thunder” CFP2000 has higher data cache miss rates However, CINT2000 is used nearly twice as much as CPF2000 for memory-hierarchy oriented papers