2. Hypergraph Partitioning for VLSI CAD: Methodology for Heuristic Development, Experimentation and Reporting Andrew E. Caldwell, Andrew B. Kahng, Andrew A. Kennings* and Igor L. Markov UCLA Computer Science Department {caldwell,abk,imarkov}@cs.ucla.edu *Cypress Semiconductor, Inc. ank@cypress.com Supported by Cadence Design Systems, Inc.

3. Overview F Risks in application-driven research F Methodology for research, experiment design –context and use-model –high-quality experimental testbed –these are real needs! F Methodology for reporting –compare with the best –compare in a useful way –well-addressed in metaheuristics community F Conclusions

4. Application-Driven Research F Well-studied areas have complex, "tuned" metaheuristics F Risks of poor research methodologies –irreproducible results or descriptions –no enabling account of key insights underlying the contribution –experimental evidence not useful to others u inconsistent with driving use model u missing comparisons with leading-edge approaches –Let’s look at some requirements this induces...

5. Awareness of Use-Model Context F Example: hypergraph bipartitioning F Runtime requirements driven by top-down global placement application F Approx 1min/20k cells for entire placement F Implied partitioner runtime: 1M nodes in several seconds F 100-1000x slower approaches not useful F Fixed terminals, node/edge weights, sparse topologies...

6. Robust Experimental Testbed F Good baseline implementations needed –“improvements” may look good only when applied to poor heuristics/implementations F Details matter –component based implementation allow fine- grained comparisons –reference implementations are needed

7. Application-Driven Research F Well-studied areas have complex, "tuned" metaheuristics F Risks of poor research methodology F Risks of poor reporting methodology –incomparable reporting styles –inappropriate benchmark usage (e.g., wrt use model)

8. Methodologies for Reporting F Well studied in the metaheuristics community (Gent94, Barr95) F Algorithm descriptions must be sufficient to allow reproduction –Benchmark implementations are a big help (e.g., Dutt-Deng LIFO FM) F Appropriate comparison of metaheuristics

9. Example Best-so-far Reporting Alg 1 Alg 2 Alg 3 Runtime  Cut 

10. Application-Driven Research F Well-studied areas have complex, "tuned" metaheuristics F Risks of poor research methodology F Risks of poor reporting methodology F Pitfalls –winners are not clear –reported improvements spurious or difficult to apply

11. Application-Driven Research F Well-studied areas have complex, "tuned" metaheuristics F Risks of poor research methodology F Risks of poor reporting methodology F Pitfalls F These risks are real !!!

12. Illustration of Pitfalls F Sample implicit decisions in FM –implicit =underspecified/not described –tie-breaking in move selection u in the direction last moved (towards) u opposite to the direction last moved (away) u always to partition0 (part0) –gain update options  update all adjacent nodes (all  Gains)  update only nodes with non-zero  gains (nonzero)

13. Effect of Implicit Decisions F Stunning average cutsize difference for flat partitioner with worst vs. best combination –far outweighs “new improvements”

14. Effect of Implicit Decisions F One wrong decision can lead to misleading conclusions w.r.t. other decisions –“part0” better than “toward” with nonzero  gain updates –worse when all adjacent nodes are updated

15. Effect of Implicit Decisions F Stronger optimization engines mask flaws –ML CLIP > ML LIFO > Flat CLIP > Flat LIFO –less dynamic range  ML masks bad flat implementation

16. Pitfall of Poor Implementation F Compare 2 implementations of CLIP-FM –15X difference in solution quality!! F Clear need for benchmark implementations

17. Conclusions F Work with mature heuristics requires mature methodologies F Identified research methodology risks F Identified reporting methodology risks F Community needs to adopt standards for both –reference “benchmark” implementations –vigilant awareness of use-model and context –reporting method that facilitates comparison