METRICS Standards and Infrastructure for Design Productivity Measurement and Optimization Andrew B. Kahng and Stefanus Mantik UCLA CS Dept., Los Angeles, CA DAC 2000 Birds-of-a-Feather Meeting June 7, 2000

Meeting Agenda Welcome Introduction and Motivations for METRICS METRICS system architecture  Discussion: Requirements for standard METRICS system architecture METRICS standards  Discussion: Potential standard METRICS names/semantics Open Discussion Conclusion: Action Items going forward

Motivations How do we improve design productivity ? Does our design technology / capability yield better productivity than it did last year ? How do we formally capture best known methods, and how do we identify them in the first place ? Does our design environment support continuous improvement of the design process ? Does our design environment support what-if / exploratory design ? Does it have early predictors of success / failure? Currently, there are no standards or infrastructure for measuring and recording the semiconductor design process

Purpose of METRICS Standard infrastructure for the collection and the storage of design process information Standard list of design metrics and process metrics Analyses and reports that are useful for design process optimization METRICS allows: Collect, Data-Mine, Measure, Diagnose, then Improve

Potential Data Collection/Diagnoses What happened within the tool as it ran? what was CPU/memory/solution quality? what were the key attributes of the instance? what iterations/branches were made, under what conditions? What else was occurring in the project? spec revisions, constraint and netlist changes, … User performs same operation repeatedly with nearly identical inputs  tool is not acting as expected  solution quality is poor, and knobs are being twiddled

Benefits Benefits for project management  accurate resource prediction at any point in design cycle up front estimates for people, time, technology, EDA licenses, IP re-use...  accurate project post-mortems everything tracked - tools, flows, users, notes no “loose”, random data or information left at project end (no more log files!!!)  Management console web-based, status-at-a-glance of tools, designs and systems at any point in project Benefits for tool R&D  feedback on the tool usage and parameters used  improve benchmarking

Current Status Complete prototype of METRICS with industry standard components (Oracle, Java, XML, HTML, etc.) Metricized place and route runs on 100+ designs; seeking access to Synopsys regression suite data Complete metricization of Cadence system-level timing flow Metricization of synthesis and Verilog simulation tools Initial feedback from industry on METRICS standards Attempting to spec Intel requirements for METRICS system

METRICS System Architecture

Inter/Intra-net DB Metrics Data Warehouse Web Server Java Applets Data Mining Reporting Transmitter wrapper Tool Transmitter API XML

Transmitter Wrapper-based Perl scripts that wrap log files and STDOUT Use existing log files (minor or no change in tool codes) Completely dependent on log files Metrics list is limited to the available reported data Need extra process API-based C/C++ library that is embedded inside tools Does not depend on log files Data are obtained directly from tools Require some changes in tool codes

METRICS Performance Transmitter  low CPU overhead multi-threads / processes – non-blocking scheme buffering – reduce number of transmissions  small memory footprint limited buffer size Reporting  web-based platform and location independent  dynamic report generation always up-to-date example: correlation plot – understand the relation between two metrics and find the importance of certain metrics to the flow

Example Reports hen 95% rat 1%bull 2% donkey 2% % aborted per machine % aborted per task BA 8% ATPG 22% synthesis 20% physical 18% postSyntTA 13% placedTA 7% funcSim 7% LVS 5% LVS convergence time LVS %

Current Results CPU_TIME = NUM_CELLS (corr = 0.93) More plots are accessible at 

COM-Based Alternative EDA tools:  provide COM interface for capturing internal information  add information collections (“counters”) inside the tools METRICS transmitter:  get information via the COM interface  format the data in XML, encrypt the message, and send it to the server Benefit:  allow independent development for transmitter and tools

Discussion on METRICS Architecture

METRICS Standards

Standard metrics naming across tools  same name  same meaning, independent of tool supplier  generic metrics and tool-specific metrics  no more ad hoc, incomparable log files Standard schema for metrics database Standard middleware for database interface For complete current lists see:

Examples of Metrics Partial list of metrics being collected now in Oracle8i

Levels of Metrics Tool/Process level Project level Enterprise level

METRICS Site Complete list of proposed metrics Source codes for METRICS server and API List of presentation on METRICS Link to various sites related to METRICS

Discussion on METRICS Standards