Paper Review: Hole Analysis For Functional Coverage Data by Oded Lachish, Eitan Marcus, Shmuel Ur, Avi Ziv @IL.IBM

1
Paper ReviewHole Analysis For Functional
Coverage Databy Oded Lachish, Eitan Marcus,
Shmuel Ur, Avi Ziv _at_IL.IBM

• ?????? ????? 0-32069023
• ?????? ??????'??? 21052137-3
• CAD of VLSI Systems (046880)
• 21/01/03

2
Functional Coverage

• Functional Coverage -- A means of
• Measuring the efficiency of Functional
  Verification.
• Finding how and in what directions to improve
  Test Generation in order to increase coverage.
• And as a result, reach more corner verification
  cases and CATCH MORE BUGS !!!
• Compare to older, less effective coverage methods
  (toggle, line, cross-state), Functional Coverage
  models closely the design functionality.

3
Functional Coverage Motivation

• Just testing is not enough
• Manual testing (MT) time-consuming and only gives
  us relatively simple cases.
• Efficient bug-finding with random testing (RT),
  but RT machine-resource consuming and needs to be
  tracked closely for efficiency with a dedicated
  tool
• The 1M question of verification what level of
  coverage are we achieving?! Without dedicated
  tool, cannot answer this.

4
Functional Coverage in a Nutshell

• Define list (space) of complete functionality of
  design. Comprises of verification tasks i.e,
  cases we expect to reach.
• Track testing results (mostly RT, also MT) and
  check which tasks were reached. (Groups of) tasks
  not reached are referred to as holes.
• In order to plug the holes
• Change/improve test-generator or
  testing-environment as needed or,
• Define restriction on coverage model if hole is
  not real case.

5
Functional Coverage Cntd.

• An Iterative ProcessResume testing to check
  improvement in total tasks covered ?report
  coverage again
• Cases-reached / Total-Case-Space Coverage

6
Functional Coverage An Iterative Process
7
Example Coverage ModelA Network Switch Receive
Port

• Define attribute groups. Rx-port consists of
  ltPacket received, Ports action Decisiongt pairs.
• Define attributes, for each attr. its possible
  values

8
Receive Port Packet Received
Values Attribute
Ethernet, IP-over-Eth, Layer4-o-IP-o-Eth, Eth-Control, Pkt Type
too short(0-63byte), small(64-127), medium(128-511), large(512-1023), extra-large(1024-1534), too long(1535 byte and up) Pkt Length
unicast, multicast, broadcast, unknown (destination only or source too) Layer2 Cast
intact, wrong, missing? CRC in
9
Receive Port Decision
Values Attribute
reject, own device, other device, CPU, multicast/broadcast Forward to
accept, length too short/long, CRC, collision, rx error reject reason
pass as received, append, recalculate CRC out
10
Hole Analysis

• Concise presentation of coverage data crucial for
  effective analysis of coverage results.
• Mere list of tasks not reached all but
  meaningless!
• Group together multiple single-element
  holes,Projected Holes n-dimensional holes. ?
  More significance and clarity than mere list of
  uncovered tasks.
• Case space lttype,Length,cast,CRCin,
  Fwd,Reject,CRCoutgt has N7 dimensions.
• 0-dimensional hole one point in case-space not
  reachedltIP,64-127,MC,intact,own dev
  port5,reject,CRCgt didnt happen.
• n-dimensional hole no event reached for any of
  the cases whereN-n1 attributes were at a fixed
  valueltIPoEth,,,,reject,,gt ? hole of
  n5, no IP pkts were rejected for any reason!
• Group together 0-dim. holes to more significant
  n-dim. holes.

11
Hole Analysis Coverage Data Presentation

• Aggregated Holes
• Group holes together based on Hamming Distance
  1. (Hamming dist. of different attributes).
  Example with 2 attributes (1-covered, 0-uncovered)

Rej collision Rej on Length Rej on Bad CRC Accept
1 1 0 0 port0 owndev
0 0 1 1 port1 owndev
0 0 1 1 port2 owndev
1 1 0 1 Other dev Other dev
1 1 1 1 CPU CPU
? better Partitioned Holes.
12
Conclusions

• Coverage emerging as critical tool in verif.
  flow
• Close tracking of testing environment
  effectiveness, closely linked to RT.
• Assessment (in ) of degree of above
  effectiveness, gives a feel of how much more
  verif. needs to be done.
• Considering ever-increasing sizecomplexity of
  todays industry designs, keeping verif. up to
  the task without diverseinformative coverage
  tool next to impossible.
• Diverseinformative in-depth Hole-Analysis,
  smart grouping of holes to serve for the verif.
  engineer.

13
Food for Thought

• Modeling of functional coverage closely coupled
  to many subjects studied in the course
• Data structures use BDD for compact
  representation of sets attributes.
• Checking whether a task was covered ?
  satisfiability.
• Coverage and Formal-Verification Two Sisters
• Formal state a rule ? check for its tautology.
• Coverage state a rule in from of task didnt
  happen ? check that it is unsatisfied.
• ) No mumbo-jumbo clip-art or meaningless
  graphics were used in the making of this paper
  review. Technion, Israel, January 2003. A.E, L.F.