Design Cost Modeling and Data Collection Infrastructure Andrew B. Kahng and Stefanus Mantik* UCSD CSE and ECE Departments (*) Cadence Design Systems, Inc.

ITRS Design Cost Model Engineer cost/year increases 5% / year ($181,568 in 1990) EDA tool cost/year (per engineer) increases 3.9% / year Productivity due to 8 major Design Technology innovations  RTL methodology  …  Large-block reuse  IC implementation suite  Intelligent testbench  Electronic System-level methodology Matched up against SOC-LP PDA content:  SOC-LP PDA design cost = $20M in 2003  Would have been $630M without EDA innovations

SOC Design Cost

Outline Introduction and motivations METRICS system architecture Design quality metrics and tool quality metrics Applications of the METRICS system Issues and conclusions

Motivations How do we improve design productivity ? Is our design technology / capability better than last year? How do we formally capture best known methods, and how do we identify them in the first place ? Does our design environment support continuous improvement, exploratory what-if design, early predictions of success / failure,...? Currently, no standards or infrastructure for measuring and recording the semiconductor design process  Can benefit project management accurate resource prediction at any point in design cycle accurate project post-mortems  Can benefit tool R&D feedback on tool usage and parameters used improved benchmarking

Fundamental Gaps Data to be measured is not available  Data is only available through tool log files  Metrics naming and semantics are not consistent among different tools We do not always know what data should be measured  Some metrics are less obviously useful  Other metrics are almost impossible to discern

Purpose of METRICS Standard infrastructure for the collection and the storage of design process information Standard list of design metrics and process metrics Analyses and reports that are useful for design process optimization METRICS allows: Collect, Data-Mine, Measure, Diagnose, then Improve

Outline Introduction and motivations METRICS system architecture  Components of METRICS System  Flow tracking  METRICS Standard Design quality metrics and tool quality metrics Applications of the METRICS system Issues and conclusions

METRICS System Architecture Inter/Intra-net DB Metrics Data Warehouse Web Server Java Applets Data Mining Reporting Transmitter wrapper Tool Transmitter API XML DAC00

Wrapper-based Transmitter Example #!/usr/local/bin/perl -w $TOOL = $0; $PID = `initProject`; $FID = `initFlow -pid ${PID}`; $TID = `initToolRun -pid ${PID} -fid ${FID}`; system “sendMetrics TOOL_NAME ${TOOL}\ STRING”; … while( ) { … system “sendMetrics ${NAME} ${VALUE}\ ${TYPE}”; … } system “terminateToolRun”; system “terminateFlow -pid ${PID} -fid ${FID}”; system “terminateProject -pid ${PID}”; exit 0; Tool Log/Output Files Stream Parsers XML Encoder Encryptor Transmitter Parsers stdout stderr stdin Internet/Intranet

API-based Transmitter Example #include “transmitter.h” int main(int argc, char* argv[]) { Transmitter MTR; MTR.initProject(); MTR.initFlow(); MTR.initToolRun(); MTR.sendMetrics(“TOOL_NAME”, argv[0],\ “STRING”); … MTR.sendMetrics(Name, Value, Type); … MTR.terminateToolRun(); MTR.terminateFlow(); MTR.terminateProject(); exit 0; } stdin Internet/Intranet API Library Tool stdout stderr XML Encoder Encryptor Transmitter SendMetric

METRICS Server DB JDBC Decryptor XML Parser Java Beans Receiver Apache + Servlet Input Form Receiver Servlet Reporting Servlet External Interface Dataminer Data translator Internet/Intranet

Example Reports nexus4 95% nexus10 1% nexus11 2% nexus12 2% % aborted per machine % aborted per task BA 8% ATPG 22% synthesis 20% physical 18% postSyntTA 13% placedTA 7% funcSim 7% LVS 5% CPU_TIME = NUM_CELLS Correlation = 0.93

METRICS Performance Transmitter  low CPU overhead multi-threads / processes – non-blocking scheme buffering – reduce number of transmissions  small memory footprint limited buffer size Reporting  web-based platform and location independent  dynamic report generation always up-to-date

Flow Tracking Task sequence: T 1, T 2, T 1, T 2, T 3, T 3, T 3, T 4, T 2, T 1, T 2, T 4 S T1T1 T2T2 F T1T1 T2T2 T3T3 T3T3 T3T3 T4T4 T2T2 T1T1 T2T2 T4T4

Testbeds: Metricized P&R Flow METRICSMETRICS Placed DEF QP ECO Legal DEF Congestion Map WRoute Capo Placer Routed DEF CongestionAnalysis Incr WRoute Final DEF LEF DEF Placed DEF QP Pearl QP Opt CTGen Incr. Routed DEF WRoute Optimized DEF LEF GCF,TLF Clocked DEF Constraints Synthesis & Tech Map Pre-placement Opt GRoute QP Post-placement Opt WRoute Ambit PKS UCLA + Cadence flow Cadence PKS flow Cadence SLC flow

METRICS Standards Standard metrics naming across tools  same name  same meaning, independent of tool supplier  generic metrics and tool-specific metrics  no more ad hoc, incomparable log files Standard schema for metrics database Standard middleware for database interface

Generic and Specific Tool Metrics Partial list of metrics now being collected in Oracle8i

Open Source Architecture METRICS components are industry standards  e.g., Oracle 8i, Java servlets, XML, Apache web server, PERL/TCL scripts, etc. Custom generated codes for wrappers and APIs are publicly available  collaboration in development of wrappers and APIs  porting to different operating systems Codes are available at:

Outline Introduction and motivations METRICS system architecture Design quality metrics and tool quality metrics Applications of the METRICS system Issues and conclusions

Tool Quality Metric: Behavior in the Presence of Input Noise [ISQED02] Goal: tool predictability  Ideal scenario: can predict final solution quality even before running the tool  Requires understanding of tool behavior Heuristic nature of tool: predicting results is difficult Lower bound on prediction accuracy: inherent tool noise Input noise  "insignificant" variations in input data (sorting, scaling, naming,...) that can nevertheless affect solution quality Goal: understand how tools behave in presence of noise, and possibly exploit inherent tool noise

Monotone Behavior Monotonicity  monotone solutions w.r.t. inputs Parameter Quality Parameter Quality

Smooth Behavior Smoothness  “similar” solutions after  perturbation Solution space

Monotonicity Studies OptimizationLevel: 1(fast/worst) … 10(slow/best) Opt Level QP WL QP CPU WR WL Total CPU Note: OptimizationLevel is the tool's own knob for "effort"; it may or may not be well-conceived with respect to the underlying heuristics (bottom line is that the tool behavior is "non-monotone" from user viewpoint)

Noise Studies: Random Seeds 200 runs with different random seeds  ½-percent spread in solution quality due to random seed -0.05%

Noise: Random Ordering & Naming Data sorting  no effect on reordering Five naming perturbation  random cell names without hierarchy (CR) E.g., AFDX|CTRL|AX239  CELL00134  random net names without hierarchy (NR)  random cell names with hierarchy (CH) E.g., AFDX|CTRL|AX129  ID012|ID79|ID216  random net names with hierarchy (NH)  random master cell names (MC) E.g., NAND3X4  MCELL0123

Noise: Random Naming (contd.) Wide range of variations (±3%) Hierarchy matters % Quality Loss Number of Runs

Noise: Hierarchy Swap hierarchy  AA|BB|C03  XX|YY|C03  XX|YY|Z12  AA|BB|Z12 % Quality Loss Number of Runs

Noise is not Additive Noise 1 + Noise 2 = (Noise 1 & Noise 2 ) Cr Nr ?

Outline Introduction and motivations METRICS system architecture Design quality and tool quality Applications of the METRICS system Issues and conclusions

Categories of Collected Data Design instances and design parameters  attributes and metrics of the design instances  e.g., number of gates, target clock frequency, number of metal layers, etc. CAD tools and invocation options  list of tools and user options that are available  e.g., tool version, optimism level, timing driven option, etc. Design solutions and result qualities  qualities of the solutions obtained from given tools and design instances  e.g., number of timing violations, total tool runtime, layout area, etc.

Three Basic Application Types  Design instances and design parameters  CAD tools and invocation options  Design solutions and result qualities Given  and , estimate the expected quality of   e.g., runtime predictions, wirelength estimations, etc. Given  and , find the appropriate setting of   e.g., best value for a specific option, etc. Given  and , identify the subspace of  that is “doable” for the tool  e.g., category of designs that are suitable for the given tools, etc.

Estimation of QP CPU and Wirelength Goal:  Estimate QPlace runtime for CPU budgeting and block partition  Estimate placement quality (total wirelength) Collect QPlace metrics from regression logfiles Use data mining (Cubist 1.07) to classify and predict, e.g.: Rule 1: [101 cases, mean 334.3, range 64 to 3881, est err 276.3] if ROW_UTILIZATION <= then CPU_TIME = ROW_UTILIZATION + 55 NUM_ROUTING_LAYER - 14 NUM_LAYER Rule 2: [168 cases, mean 365.7, range 20 to 5352, est err 281.6] if NUM_ROUTING_LAYER <= 4 then CPU_TIME = NUM_ROUTING_LAYER ROW_UTILIZATION - 49 NUM_LAYER Rule 3: [161 cases, mean 795.8, range 126 to 1509, est err ] if NUM_ROUTING_LAYER > 4 and ROW_UTILIZATION > then CPU_TIME = ROW_UTILIZATION + 55 NUM_ROUTING_LAYER - 14 NUM_LAYER Data mining limitation  sparseness of data

Cubist 1.07 Predictor for Total Wirelength

Optimization of Incremental Multilevel FM Partitioning Motivation: Incremental Netlist Partitioning Scenario: Design changes (netlist ECOs) are made, but we want the top-down placement result to remain similar to previous result RefinementClustering

Multilevel Partitioning RefinementClustering

Multi-level Partitioning Multi-level FM (MLFM)  Much better than “flat” FM, very good solutions in near-linear time  Critical to performance of top-down placers Implementation choices in MLFM  V-cycling (“iterative ML”) - Karypis et al., DAC’97 a method of using an initial solution avoids clustering vertices that are in different sets allows us to run ML to improve results of previous runs  Top-level partitioning with small tolerance first partition top level with lax tolerance use the result as initial solution for another FM run decrease tolerance to what it should be, run FM  New clustering methods Tuning (solution pool size, un/coarsening ratios, etc.) Locally optimized implementations look very different !!!

Optimization of Incremental Multilevel FM Partitioning Motivation: Incremental Netlist Partitioning Scenario: Design changes (netlist ECOs) are made, but we want the top-down placement result to remain similar to previous result Good approach [CaldwellKM00]: “V-cycling” based multilevel Fiduccia-Mattheyses Our goal: What is the best tuning of the approach for a given instance? break up the ECO perturbation into multiple smaller perturbations? #starts of the partitioner? within a specified CPU budget?

Optimization of Incremental Multilevel FM Partitioning (contd.) Given: initial partitioning solution, CPU budget and instance perturbation (  I) Find: number of stages of incremental partitioning (i.e., how to break up  I ) and number of starts  T i = incremental multilevel FM partitioning  Self-loop  multistart  n  number of breakups (  I =  1 +  2 +   n ) S T1T1 F T2T2 T3T3 TnTn...

Flow Optimization Results If (27401 < num edges  34826) and ( < cpu time  ) and (perturbation delta  0.1) then num_inc_stages = 4 and num_starts = 3 If (27401 < num edges  34826) and (85.27 < cpu time  ) and (perturbation delta  0.1) then num_inc_stages = 2 and num_starts = 1... Up to 10% cutsize reduction with same CPU budget, using tuned #starts, #stages, etc. in ML FM

Outline Introduction and motivations METRICS system architecture Design quality and tool quality Applications for METRICS system Issues and conclusions

METRICS Deployment and Adoption Security: proprietary and confidential information cannot pass across company firewall  may be difficult to develop metrics and predictors across multiple companies Standardization: flow, terminology, data management Social: “big brother”, collection of social metrics Data cleanup: obsolete designs, old methodology, old tools Data availability with standards: log files, API, or somewhere in between? “Design Factories” are using METRICS

Conclusions METRICS System : automatic data collection and real- time reporting New design and process metrics with standard naming Analysis of EDA tool quality in presence of input noise Applications of METRICS: tool solution quality estimator (e.g., placement) and instance-specific tool parameter tuning (e.g., incremental partitioner) Ongoing works:  Construct active feedback from METRICS to design process for automated process improvement  Expand the current metrics list to include enterprise metrics (e.g., number of engineers, number of spec revisions, etc.)

Thank You