Coverage Analysis

1
Coverage Analysis

• By Alon Kartin
• CS Department
• Haifa University
• 3-November-2002

2
Outline

• Introduction to coverage Analysis
• Definition
• Goals
• Risks
• Types
• Comet
• Advantages and Disadvantages
• Comet Structure
• Coverage Process with Comet
• Conclusions

3
Introduction to Coverage Analysis

• What is Coverage Analysis?
• Definitions
• Coverage Goals
• Risks in Coverage
• Program based coverage
• Functional coverage

4
What is Coverage Analysis?

• Testing
• Plays major rule in the development process
• Based on tests which measures the system
• The problem
• We cant run all the possible tests
• But we need to know that all areas of design
  are tested
• The solution Test Coverage analysis is the main
  technique for demonstrating that the testing has
  been thorough

5
What is Coverage Analysis? (cont)

• Coverage definitionCoverage is any metric which
  helps to approximate the ideal by acting as
  heuristic measures that quantify verification
  completeness, and identifying inadequately
  exercised design aspects and guiding inputs
  stimulus generation
• How is it done?
• Creation list of tasks (testing requirements)
• Checking that each task is covered during the
  testing process

6
Definitions

• Coverage task - A Boolean function on a test
• The branch to the sort function was taken
• The instructions in two stages of a pipe are ADD
  and DIV
• Coverage model - A set of coverage tasks
  
• Execution of each statement in the program
• All combinations of instructions in the pipe
• Illegal task - Coverage tasks that should not
  occur
• (fix point instruction in a
  floating point pipe)
• Task Coverage List (TCL) - A list of all the
  legal tasks in a model

7
Coverage Goals

• Measure the quality of testing
• Find new testing requirements that have been
  overlooked in the test plan Example Sort
  function
• Better understanding of the design
• Forming small yet comprehensive regression suits

8
Goal 4 - Regression suits

• Regression Any repetition of tests intended to
  show that the design behavior is unchanged
• Contradictory requirements
• Should be small, so it will not be hard to
• execute it after every design change
• Should be comprehensive, so that it will catch
  all the new bugs that were introduced
• Coverage can help in creating good
• regression suites
• Set of tests that cover all the tasks that have
  been covered
• Create the smallest of such a set

9
Risks in coverage

• 1. Control flow models are ill to deal with
  missing code
• Example case statement
• Possible solution Use several coverage models
• 2. Setting low coverage goals
• Lower coverage target (50-90)is not useful
• Possible solution use simpler coverage models
  with high coverage goals than more complex models
  with lower coverage goals
• 3. A common misconception
• the testing methodology should be
  decided on an
• appropriate coverage metric and then
  generate a
• set of tests that covers it
• Example Analogy to wall painting.

10
Two types of coverage

• Program based coverage
• Functional coverage

11
Program-based coverage

• Measures the execution of tests against the
  source code of the program
• Concentrates on measuring syntactic properties in
  the execution, like
• Each statement was executed
• Each branch was taken

12
Program-based advantages

• Available for many tools
• Simplicity of use
• No special preparations
• Feedback from the tool is easy to understood
• Usually easy to measure

13
Program-based disadvantages

• Dont understand the application domain
• Hard to tune the tools to areas which the user
  think are of significant

14
Functional Coverage

• Focuses on the functionality of the program
• Design and implementation specific
• Currently, implemented manually or by using
  custom made tools
• Functional coverage models involve looking at
  several properties in parallel

15
Disadvantages

• Lack of automation
• More costly to measure
• The parts which are not of special
• concern could be neglected.

16
Program-Based Vs. Functional
17
Introduction to Coverage

• What is Coverage Analysis?
• Definitions
• Coverage Goals
• Risks in Coverage
• Program based coverage
• Functional coverage

18
Functional Coverage Example- Parcel Sorting
System

• System definition
• Design requirements
• Model definition
• Coverage process

19
Parcel Sorting System-System definition

• Developed at IBMs Haifa research laboratory
• Contains 6 PowerPC multi processor system
• How does it works?
• The top surface of the parcel is scanned,
  dimensions are measured and lenses are focused
• Reading the address
• The weight of the parcel is acquired

20
(No Transcript)
21
Parcel Sorting System- Design requirements

• The coverage model checks that
• Every type of parcel size and every
• type of address is encountered
• Every processor internally processed
• the parcel in any way
• The process can correctly interrupted
• at any stage

22
Parcel Sorting System- Coverage model definition

• Guidelines
• Choose the correct type of coverage model to
  measure coverage
• Choose coverage model to areas which the user
  think are significant
• Choose coverage model according to the testing
  resources

23
Attribute list for parcel sorting system model

• Attribute list
• (Height, Width, Length, Master, Num_processors,
  Num_addresses, Flush, Finish, Success)
• Attributes description and values
• Height (Height of parcel in milimiters) /1001
• Widht (Widht of parcel in milimiters) /1001
• Length (Lenght of parcel in milimiters) /1001
• Master 1 if parcel processed by master else 0
• Num_processors Number of busy processors when
  the parcel started (1-6)
• Num_adressess Number of addresses are on the
  parcel (0-4)
• Flush Name of the stage prior to Flush
• Finish Name of the stage prior to Finish
• Success- confidence above 0.8 in recognition

24
Model hierarchy for parcel sorting system

• Coverage models are usually built in a
  hierarchical way
• Start with small models that examine specific
  areas
• Later on, merge the small models to create larger
  models

25
Model hierarchy for parcel sorting system

• Model 1 Parcel Size (Height, Width, Length)
• Model 2 Number of addresses - (Num_addresses)
• Model 3 Load - (Master, Num_processors)
• Model 4 Messages - (Flush, Finish)
• Model 5 Success (Success)
• Model 6 All Parcel types Model 1 Model 2
• Model 7 Load and Messages Model 3 Model 4
• Model 8 Messages and Success Model 4 Model
  5
• Model 9 Complete model Model 6 Model 7
  Model 5

26
Parcel Sorting System- Coverage Process

• Restrictions on the inputs
• Examples
• There are no more than six busy processors at one
  time
• When the parcel is sent to the master, all the
  slaves are busy
• The process of finding restrictions is an
  iterative process
• The test begin - first on simple models
• Data Collection for each parcel

27
Parcel Sorting System- Coverage Process (cont)

• Looking for coverage holes
• Possible reasons
• Tests are missing
• Bugs in the application
• Bugs in the coverage process
• Missing restrictions
• The process is being iterated until
• Recourses or time are exhausted
• Coverage is complete

28
Huge number of tasks

• In many cases the number of tasks seems huge, in
  the parcel sort system the number of the tasks
  is
• 6662657721,375,920
• In practice, those kind of numbers are not a
  problem
• Restrictions usually reduce the numbers of the
  tasks by a number of orders
• We start with partial models which have a small
  number of tasks, and not with the full model.

29
Introduction to Coverage

• What is Coverage Analysis?
• Definitions
• Coverage Goals
• Risks in Coverage
• Program based coverage
• Functional coverage

30
Comet

• Why Comet?
• Advantages and Disadvantages
• Comet Structure
• Coverage Process with Comet

31
CometCOverage MEasurement Tool

• General Purpose Tool for User
• Defined Coverage Models

32
Reminder

• The problems of functional coverage
• Be specific but, most commercial tool are
  generic.
• Two possible options
• Do coverage manually
• Time consuming
• Limited size of models
• Build a new specific coverage tool
• Large development effort
• Tend to be error-prone

33
The Comet solution

• Most features of coverage tools are common
• to all tools and independent of the coverage
  models
• Separation of the coverage models from the tools
• It provides all the functionality of existing
  tools such as data gathering, coverage reports,
• AND provide both generic and specific
  coverage models

34
Advantages of the Comet

• The user can define model, according to his
  specific needs easily
• Use only one coverage tool, instead of buying set
  of tools one for each coverage model
• Allows sharing of coverage models between
  projects
• Adapting the coverage models to the testing
  resources
• Refining the models during testing

35
Disadvantages of the Comet

• Complexity of model definition
• Low performance compared with dedicated tools

36
Comet Structure
Comet
Relational Database
37
Comet structure

• General purpose coverage tool
• Built on top of a relational database
• Database provides stability
• SQL used as model definition language
• Inputs
• Definition of coverage models described with
  logic predicates (and, not, for, after, etc.) and
  simple arithmetic
• Event traces
• Outputs
• Detection of illegal tasks
• Coverage reports
• Coverage status - what is covered
• Coverage progress
• Regression suites with coverage properties

38
Comet structure

• Insertion engine insert events from event traces
  into the database
• Processing engine process the traces in the
  database in order to detect coverage tasks
  according to the model definition
• Coverage analyzer analyze the measurement
  results and prepare coverage analysis reports

39
The Coverage Process with Comet
Tests
40
Examples of Coverage Models
41
Coverage model definition

• The process
• Defining the attributes of the model
• Specifying how to extract the attributes from the
  event trace
• Separating between the legal and illegal tasks of
  the model

42
Coverage model definition

• Step 1 Define the attributes of the coverage
  model and how to extract them from the event
  trace
• The SELECT part of an SQL query on the event
  trace
• For the IIwI model

43
Coverage model definition

• Step 2 Specify how to extract tasks from the
  event trace
• The WHERE part of an SQL query on the event trace
• For the IIwI model
• -- Correct order of eventse1.InstOrder lt
  e2.InstOrder and e2.InstOrder lt e3.InstOrder
  and(e3.InstOrder - e1.InstOrder) lt 6 and
  e1.InstOrder gt 5 and
• -- No interrupt in first and second instructions,
  
• -- interrupt between theme1.InterruptBasic
  'NoIntr' and e2.InterruptBasic ltgt 'NoIntr' and
  e3.InterruptBasic 'NoIntr'

44
Coverage Model Definition

• Step 3 Specify restrictions on legal tasks
• The WHERE part of an SQL query on the TCL
• For the branch unit pipe model
• -- No branch can pass another in the pipe
  stages( (a1lt b1) or (b1lt c1) or (c1lt d1) or
  (d1lt e1) or (e1lt f1) or (f1lt g1) )
• -- Surprise Branches may only be in stages
  5-9(a2 0 and a1 between 1 and 4) or(b2 0
  and b1 between 1 and 4) or(c2 0 and c1 between
  1 and 4) or(d2 0 and d1 between 1 and 4)

45
Data Collection

• Generate coverage traces
• From test programs (example IIwI model)
• From simulation traces (example branch unit pipe
  model)
• Insert the traces to the event trace table
• Automatic insertion from simulator
• Manual or semi-automatic insertion
• Measure coverage on inserted traces
• Apply model query on the trace to find all
  coverage tasks in the trace
• Identify illegal tasks
• Increase coverage counter for legal tasks found
  in the trace

46
Detection of Illegal Events

• A coverage task that is in the event trace but
  not in the TCL is considered illegal
• Example in the branch unit pipe a1 4 and
  a2 0
• Violates second restriction
• Branch in stage 4 cannot be surprise branch
• Information on where the illegal event was found
  is stored in Comet
• Trace where the violation occurred
• Cycle in which it occurred
• Restrictions that were violated
• User is notified upon detection of illegal event

47
Coverage Status Reports

• Examine the coverage map in a specific cross
  section
• Used to detect holes in the coverage
• Improve test generation to cover uncovered areas
• Help finding new restrictions

48
Coverage Progress Reports

• Show progress of coverage over time
• Detects effects of changes in testing strategy
• Can find when testing ran out of steam

49
Conclusions

• Comet separates coverage models from coverage
  tool
• Comet enables deployment of coverage in new
  places that were not accessible previously
• More than a hundred special coverage models which
  fitting specific applications implemented
• Helps designer to understand the design better

50
Comet

• Why Comet?
• Advantages and Disadvantages
• Comet Structure
• Coverage Process with Comet

51
10 Things about coverage

• No resources No coverage
• Coverage not just a principle
• Set high Coverage goals
• Use automatic test generator
• Make your model compact
• Effective coverage reports
• Impossible? Be there!
• Hierarchy is better
• Restrictions set them first

52

• Functional for the hard-to-achieve
• Program based for the rest