METRICS: A System Architecture for Design Process Optimization

1
METRICS A System Architecture forDesign Process
Optimization
Andrew B. Kahng and Stefanus Mantik

• Motivations
• Value of CAD tools improvement not clear
• value well-defined only in context of overall
  design process
• Design process includes other aspects not like
  any flow/methodology bubble chart
• must measure to diagnose, and diagnose to improve
• Many possibilities for what to measure
• solution record everything, then mine the data
• Unlimited range of possible diagnoses
• User performs same operation repeatedly with
  nearly identical inputs
• tool is not acting as expected
• solution quality is poor, and knobs are being
  twiddled
• On-line docs always open to particular page
• command/option is unclear

Abstract We describe the architecture and
prototype implementation of METRICS, a system
aimed at improving design productivity through
new infrastructure for design process
optimization. A key precept for METRICS is
that measuring a design process is a prerequisite
to learning how to optimize that design process
and continuously achieve maximum productivity.
METRICS, therefore, (i) gathers characteristics
of design artifacts, design process, and
communication during system development effort,
and (ii) analyzes and compares that data to
analogous data from prior efforts. METRICS
infrastructure consists of three parts (i) a
standard metrics schema, along with metrics
transmittal capabilities embedded directly into
EDA tools or into wrappers around tools (ii) a
metrics data warehouse and API for metrics
retrieval and (iii) data mining and
visualization capabilities for project
prediction, tracking, and diagnosis. Salient
aspects of METRICS include the following. First,
a standard metrics schema, along with standard
naming and semantics, allows a metric from one
tool to have the same meaning as the same metric
from another tool from a different vendor.
Second, transmittal APIs that are easily
embeddable within tools allow freedom from the
"log files" that currently provide only limited
visibility into EDA tools. With appropriate
security and access restrictions, these APIs can
prevent loss of proprietary information while yet
enabling detailed tracking of the design process.
Third, at the heart of METRICS is a centralized
data warehouse that stores metrics information.
Several means of data retrieval and visualization
(e.g., web-based project tracking and prediction)
afford user flexibility. Finally,
industry-standard components and protocols (http,
XML, Java, Oracle8i, etc.) are used to create a
robust, reliable system prototype.
METRICS System Architecture

• METRICS Reporting
• Web-based
• platform independent
• accessible from anywhere
• Example correlation plots created on-the-fly
• understand the relation between two metrics
• find the importance of certain metrics to the
  flow
• always up-to-date

• METRICS Transmitter
• No functional change to the tool
• use API to send the available metrics
• Low overhead
• example standard-cell placer using Metrics API ?
  lt 2 runtime overhead
• even less overhead with buffering
• Wont break the tool on transmittal failure
• child process handles transmission while parent
  process continues its job

Example of METRICS XML, API and Wrapper
Example of Reports

• METRICS Standards
• Standard metrics naming across tools
• same name ? same meaning, independent of tool
  supplier
• generic metrics and tool-specific metrics
• no more ad hoc, incomparable log files
• Standard schema for metrics database

lt? xml version1.0 ?gt ltMETRICSPACKETgt ltREQUESTgt
ltTYPEgt TOOL lt/TYPEgt ltPROJECTIDgt 173
lt/PROJECTIDgt ltFLOWIDgt 9 lt/FLOWIDgt ltPARAMETERgt
32 lt/PARAMETERgt lt/REQUESTgt ltMETRICSgt ltPROJECTI
Dgt 173 lt/PROJECTIDgt ltFLOWIDgt 9
lt/FLOWIDgt ltTOOLIDgt P32 lt/TOOLIDgt ltDATETIMEgt
93762541300 lt/DATETIMEgt ltNAMEgt TOOL_NAME
lt/NAMEgt ltVALUEgt CongestionAnalysis
lt/VALUEgt lt/METRICSgt lt/METRICSPACKETgt
/ API Example / int main(int argc, char
argv ) ... toolID initToolRun(
projectID, flowID ) ... printf( Hello
World\n ) sendMetric( projectID, flowID,
toolID, TOOL_NAME, Sample ) sendMetric(
projectID, flowID, toolID, TOOL_VERSION,
1.0 ) ... terminateToolRun( projectID,
flowID, toolID ) return 0
Congestion vs. WL
Abort by Task
Wrapper example ( File, PID, FID )
_at_ARGV TID initToolRun PID FID open ( IN,
lt File ) while ( ltINgt ) if (
/Begin\s(\S)\son\s(\S.)/) system
sendMetrics PID FID TID TOOL_NAME
1 system sendMetrics PID FID TID
START_TIME 2 ... system
terminateToolRun PID FID TID
LVS Convergence
Capo/Cadence Flow

• Observations from experience with a previous
  prototype
• Implemented by OxSigen LLC (Fenstermaker, George,
  Thielges) in Siemens Semicustom Highway flow
• The METRICS system must be non-intrusive. The
  best choice for the system is if it is embedded
  in the tools.
• Big brother type issues must be spelled out
  clearly at the beginning, and buyoff from user
  advocates must be considered. All data must be
  anonymized and any attempt to profile or quantify
  individual performance on a project is dangerous
  (but useful).
• There is still a very big problem with flows.
  Ideally, the flow should be standardized, with
  Makefile type build environment for batch chip
  creation. There is no obvious common way to
  handle interactive tools yet, so we must be able
  to metricize flows in a standard way (which
  requires standard flows).
• The CAD / design community must get together to
  standardize (or better educate) people on flow
  terminology, especially now that so many new
  hybrid tools are emerging which combine
  traditional flow steps. If we simply had a
  standard set of agreed upon milestones that occur
  during the lifecycle of a design, we could start
  to do accurate and more worthwhile benchmarking
  and prediction.
• There is still a very big problem with
  standardized data management (version control),
  i.e., lots of custom codes to work around source
  code control systems in real world environments.
• Project management tools need to be more
  standardized and widely used. These tools act
  like metrics transmitters for project-related
  information such as time allotted for certain
  tasks. This is critical for prediction of
  project-related details (how long to completion
  from this point, etc.).

• Conclusions and Ongoing Work
• Completion of METRICS server with Oracle8i, Java
  servlet, and XML parser
• Initial transmittal API in C
• METRICS wrapper for Cadence PR tools
• Simple reporting scheme for correlations
• Work with EDA, designer community to establish
  standards
• tool users list of metrics needed for design
  process optimization
• tool vendors implementation of the metrics
  requested with the standardized naming
• Improve the transmitter
• add message buffering
• recovery system for network / server failure
• Extend METRICS system to include project
  management tools, email communications, etc.
• Additional reports, data mining