Comparative Assessment of Software Quality Classification Techniques: An Empirical Case Study

1
Comparative Assessment of Software
QualityClassification Techniques An Empirical
Case Study

• TAGHI M. KHOSHGOFTAAR, NAEEM SELIYA
• Empirical Software Engineering Laboratory, Dept.
  of Computer Science and Engineering, Florida
  Atlantic University,
• Empirical Software Engineering, 9, 229-257, 2004.
  (G) 2004 Kluwer Academic Publishers.
  Manufactured in The Netherlands

2
Software metrics-based quality classification
models SMQC

• Predict a software module as fault-prone (fp) or
  not fault-prone (nfp)
• Using SMQC early in development helps
  cost-effectiveness
• Planned and better use of test and improve
  measures
• Common SMQC models available
• include logistic regression
• case-based reasoning
• classification and regression trees (CART)
• tree-based classification with S-PLUS
• Sprint-Sliq
• C4.5
• Treedisc

3
The R D

• Comparative evaluation of 7 classification
  techniques and/or tools
• Introduction of expected cost of
  misclassification (ECM)
• unified measure to compare the performances of
  different software quality classification models
• ECM is computed for different cost ratios using
  
• Type I Costs of nfp module misclassified as fp
  fault prone
• Type II Costs of fp module misclassified as nfp
  not fault prone
• ECM can help use the appropriate SMQC for
  specific modules which have varying likelihoods
  of being misclassified as fp or nfp.

4
The Paper

1. Introduction
2. Description classification modeling methods
3. Case study used in this paper is described
4. Modeling objective, methodology, and techniques
   employed in comparing the different
   classification models
5. Results
6. Conclusions of comparative study

5
Models

• CART - Classification and regression trees -
  decision tree system for e.g. data mining,
  automatically finds significant patterns large
  complex databases
• S-PLUS - A solution for advanced data analysis,
  data mining, and statistical modeling -
  regression tree-based models
• C4.5 algorithm - employs decision trees to
  represent a quality model
• Treedisc algorithm - constructs a regression tree
  from an input data set, that predicts a specified
  categorical response variable based on one or
  more predictors
• Sprint-Sliq builds classification tree models
• Logistic regression - statistical modeling
  technique , can be adapted for classification
• Case-based reasoning (CBR) - find solutions to
  new problems based on "cases" in a case
  library - a module currently under development
  is probably fp if a module with similar
  attributes in an earlier release (or similar
  project) was fp.

6
The Study

• 4 successive releases of large legacy
  telecommunications system
• Release 1 baseline for classification models(7)
• Measure faults after release that require code
  change
• These are costly faults as that require visits
  and repair
• A module is nfp if no post-release faults else
  fp
• Faults against LOC also calculated

7
Metrics

• Number of post release faults
• LOC
• Other Metrics

8
Conclusions

• Quality estimation models do work
• Need to evaluate models
• fp prediction is vital
• ECM was testing
• Cannot be generalised (single system)
• Unclear effect of release
• Unclear reasons for differences in performance
• New study needed with different type of software