DevCOP: A Software Certificate Management System for Eclipse

1
DevCOP A Software Certificate Management System
for Eclipse

• Mark Sherriff and Laurie Williams
• North Carolina State University
• ISSRE 06
• November 10, 2006

2
Agenda

• Background
• Motivation and Hypothesis
• Software Certificates and Software Certificate
  Management Systems (SCMS)
• DevCOP (Defect Estimation with VV Certificates
  on Programming)
• Research Goal and Methodology
• DevCOP Certificates
• The DevCOP SCMS Eclipse plug-in
• Limitations
• Current Status
• Questions

3
Motivation

• Software Reliability
• Often not estimated until development is complete
• Actual reliability not known until system is
  shipped to customers
• Corrective action is more expensive later in the
  process
• If defect density could be estimated in-process
• Steps could be taken to address issues early
• More economical, could improve development effort

4
Hypothesis

• Defect density estimation can be based upon the
  history of verification and validation techniques
  that have been performed on the project.
• Questions that need to be answered
• What is the best way to record VV techniques?
• How do I build a model that can predict defect
  density with VV information?

5
Recording VV Techniques

• Software Certificates
• A record of a verification and validation (VV)
  practice employed by developers and used to
  support traceability between code and evidence of
  the VV technique used
• Software certificates could be any type of
  record
• Logs from test case runs
• Reports from code inspections
• Details on pair programming assignments
• However, these are all different forms of VV
  information and maintaining this information
  could be expensive.

6
Recording VV Techniques

• Software Certificate Management Systems
• Provides an interface and infrastructure to
  automatically create, maintain, and analyze
  software certificates
• Benefits
• Software maintenance
• Analysis of VV technique effectiveness
• Reference in future projects
• Current research
• OGI/PSU Programatica, a SCMS for Haskell

7
DevCOP

• Research hypothesis Defect density estimation
  can be based upon the history of VV techniques
  that have been performed on the project.
• Methodology Build a parametric model using
  software certificate information and previous
  project defect data to create a prediction model
  for future projects.

8
DevCOP Certificates

• Records
• identifying information for the function it is
  associated with including its name, signature,
  class, and file location
• identifying information for the developer that
  created it
• the type of VV technique used
• a hash of the functions abstract syntax tree
  (AST) and
• a significance weight.

9
DevCOP Certificates

• Types of Certificates
• Manual
• Includes all manual techniques, such as pair
  programming and code inspections
• Automated Static Analysis
• Includes all techniques that can be run
  automatically on uncompiled code
• Dynamic
• Includes all techiques performed automatically at
  run time, such as black box testing
• Formal
• Includes all formal methods, such as lambda
  calculus and proofs

10
DevCOP SCMS Eclipse Plug-in

• Currently supports manual VV techniques and
  jcoverage certificates
• Provides different methods for examining and
  managing certificate data
• Demo

11
Potential Side Effects

• Retrospective Causal Analysis
• Once certificates are recorded on a project and
  bugs are reported, developers can use certificate
  and defect information to evaluate the efficacy
  of their VV practices.
• Building certificate information in with compiled
  code base
• If certificate information could travel with
  compiled code, it could be referenced at runtime
  so that other systems could evaluate whether it
  was to work with that system.

12
Original Hypothesis

• Parametric Modeling Method by which dependant
  variables are related to one or more independent
  variables with regard to previous data
• In Software Engineering
• Purpose is to provide an estimated answer to a
  software development question earlier in the
  development lifecycle
• Famous SE parametric models COCOMO 81 and COCOMO
  II

13
Original Hypothesis

• Early Research Software Testing Reliability
  Early Warning
• Uses a suite of metrics gathered on static code
  to provide a reliability estimate
• If a reliability estimate can be created from
  testing and static metrics, could it be improved
  if we added other verification and validation
  information to the model?

14
Limitations of DevCOP

• Eclipse Environment
• Even though numerous companies do use Eclipse, it
  might not be standardized
• Tool Selection
• DevCOP uses JCoverage, which might not be the
  approved coverage tool
• Does add minimal overhead
• Even adding one step can be too much

15
Evolution of Research

• The core theory behind DevCOP is that artifacts
  generated during software development can be
  useful during the development process.
• A certificate is simply an artifact of software
  development.
• Can we gather certificates or other artifacts
  with NO ADDITIONAL developer involvement?

16
Evolution of Research

• SDAA Software Development Artifact Analysis
• By mining source control data, testing records,
  and other VV information, new associations can
  be found between code units.
• The type of association isdetermined by the
  typeof development artifact used.

17
Uses of SDAA

• Identification of Areas of Risk
• Associations generated from defect data
• Impact Analysis
• What files are likely to change together?
• Test Case Prioritization
• How does a particular testrelate to a
  givenassociation?

18
Thank you!

• Questions? Queries? Quandries?
• Contact Information
• Mark Sherriff
• mark.sherriff_at_ncsu.edu
• DevCOP Project http//agile.csc.ncsu.edu/devcop/