Defect Density Estimation through Verification and Validation

1
Defect Density Estimation through Verification
and Validation

• Mark Sherriff and Laurie Williams
• North Carolina State University
• HCSS 06
• April 19, 2006

2
Agenda

• Background
• Motivation and Hypothesis
• Software Certificates and Software Certificate
  Management Systems (SCMS)
• DevCOP (Defect Estimation with VV Certificates
  on Programming)
• Research Goal and Methodology
• DevCOP Certificates
• The DevCOP SCMS Eclipse plug-in
• Limitations
• Current Status
• Questions

3
Motivation

• Software Reliability
• Often not estimated until development is complete
• Actual reliability not known until system is
  shipped to customers
• Corrective action is more expensive later in the
  process
• If defect density could be estimated in-process
• Steps could be taken to address issues early
• More economical, could improve development effort

4
Hypothesis

• Defect density estimation can be based upon the
  history of verification and validation techniques
  that have been performed on the project.
• Questions that need to be answered
• What is the best way to record VV techniques?
• How do I build a model that can predict defect
  density with VV information?

5
Recording VV Techniques

• Software Certificates
• A record of a verification and validation (VV)
  practice employed by developers and used to
  support traceability between code and evidence of
  the VV technique used
• Software certificates could be any type of
  record
• Logs from test case runs
• Reports from code inspections
• Details on pair programming assignments
• However, these are all different forms of VV
  information and maintaining this information
  could be expensive.

6
Recording VV Techniques

• Software Certificate Management Systems
• Provides an interface and infrastructure to
  automatically create, maintain, and analyze
  software certificates
• Benefits
• Software maintenance
• Analysis of VV technique effectiveness
• Reference in future projects
• Current research
• OGI/PSU Programatica, a SCMS for Haskell

7
Parametric Modeling

• Method by which dependant variables are related
  to one or more independent variables with regard
  to previous data
• In Software Engineering
• Purpose is to provide an estimated answer to a
  software development question earlier in the
  development lifecycle
• Famous SE parametric models COCOMO 81 and COCOMO
  II

8
Parametric Modeling

• Software Testing Reliability Early Warning
• Java and Haskell versions
• Uses a suite of metrics gathered on static code
  to provide a reliability estimate
• The model is calibrated to a particular
  organization using a regression equation
• STREW is a good option because using operational
  profiles for a similar prediction can be
  expensive to create and maintain
• If a reliability estimate can be created from
  testing and static metrics, could it be improved
  if we added other verification and validation
  information to the model?

9
DevCOP

• Research hypothesis Defect density estimation
  can be based upon the history of VV techniques
  that have been performed on the project.
• Methodology Build a parametric model using
  software certificate information and previous
  project defect data to create a prediction model
  for future projects.

10
DevCOP Certificates

• Records
• identifying information for the function it is
  associated with including its name, signature,
  class, and file location
• identifying information for the developer that
  created it
• the type of VV technique used
• a hash of the functions abstract syntax tree
  (AST) and
• a significance weight.

11
DevCOP Certificates

• Types of Certificates
• Manual
• Includes all manual techniques, such as pair
  programming and code inspections
• Automated Static Analysis
• Includes all techniques that can be run
  automatically on uncompiled code
• Dynamic
• Includes all techiques performed automatically at
  run time, such as black box testing
• Formal
• Includes all formal methods, such as lambda
  calculus and proofs

12
DevCOP SCMS Eclipse Plug-in

• Currently supports manual VV techniques and
  jcoverage certificates
• Provides different methods for examining and
  managing certificate data
• Demo

13
Building the Model
14
Limitations

• Granularity of Certificates
• Method level, not class or line of code
• Composition of Certificates
• Not much is known about how one VV technique
  adds to another
• Defect Severity
• All defects are treated equally

15
Potential Side Effects

• Retrospective Causal Analysis
• Once certificates are recorded on a project and
  bugs are reported, developers can use certificate
  and defect information to evaluate the efficacy
  of their VV practices.
• Building certificate information in with compiled
  code base
• If certificate information could travel with
  compiled code, it could be referenced at runtime
  so that other systems could evaluate whether it
  was to work with that system.

16
Thank you!

• Questions? Queries? Quandries?
• Contact Information
• Mark Sherriff
• mark.sherriff_at_ncsu.edu
• DevCOP Project http//agile.csc.ncsu.edu/devcop/