Source code metrics in the software industry

1
Source code metrics in the software industry
School of Computer and Information Science, Edith
Cowan University PhD research project by Tim
Littlefair supervised by Dr Thomas
O'Neill http//www.fste.ac.cowan.edu.au/tlittlef
2
Source code metrics

• Software metrics are generally classified as
  being divided into two categories process
  metrics and product metrics.
• The value of process metrics to aid software
  process management is now widely accepted.
• Source code metrics are a subset of the product
  metric classification. Many source metrics have
  been proposed, there is no consensus as to which
  are useful.

3
Project features

• Metrics of interest were selected.
• Automated measurement tool was implemented.
• Tool was deployed and evaluated by developers
  performing real-world development under
  commercial conditions.
• Internet was used to distribute tool and solicit
  evaluation feedback.
• Project is concluding with experiment on use of
  metrics data as support for software inspection
  process.

4
Metrics Implemented

• Procedural metrics (measured on a per-function
  basis) lines of code, lines of comment,
  McCabe's Cyclomatic Complexity.
• Metrics of object oriented design (proposed by
  Chidamber and Kemerer) depth of inheritance
  tree, number of children, coupling between
  objects, weighted methods per class.
• Structural metrics (based on work by Henry
  Kafura) fan-in, fan-out, information flow.

5
Evaluation survey

• Tool was publicised via USENET, deployed (and
  refined) over a 6 month period.
• USENET and email addresses from FTP logs used to
  publicise evaluation URL.
• 25 respondents over 3 month period, weak
  consensus on value of procedural metrics, opinion
  neutral or marginally negative on OO design
  metrics and structural metrics.

6
Review Experiment

• Survey yielded little interesting data - need to
  extend project to make it worthwhile.
• Experiment designed to attempt to refute null
  hypothesis (i.e. "metrics are of no value") by
  detecting positive effect of metrics use.
• Simulated code review - comparing performance of
  groups with and without metrics support.

7
The exercise

• 5 Java classes to be reviewed.
• Each respondent to give yes/no answer on the
  presence of each of 5 risk factors for each class
  (e.g. "Excessive length", "Inadequate
  commenting").
• Comparison of performance requires independently
  determined set of 'correct' answers.

8
Treatment groups

• Group 0 perform the exercise without metric
  support with plenty of time (1 hour)
• Group 1 perform the exercise without metric
  support under time constraint (15 mins)
• Group 2 perform the exercise with metric support
  under time constraint (15 mins)
• Group 0 are used to derive 'correct' responses,
  value of metrics support is assessed by seeing if
  group 2 reflects these better than group 1.

9
Experimental Outcomes

• Code review experiment closed after attracting 15
  volunteers (6 in group 0, 4 in group 1 and 5 in
  group 2)
• This was less than we were hoping for, but enough
  to do analysis on the results
• Derivation of correct results by selecting a
  threshold for group 0 responses
• Statistical techniques used in processing of
  returns contingency tables, receiver-operating
  characteristic analysis, chi-squared analysis.

10
Cumulative Responses

• The table below shows the number of respondents
  in each group, together with the number of
  positive (risk present) responses by each group
  to each question.
• Group 0 responses are distilled to derive the
  correct answer to each of the 25 questions, the
  performance of groups 1 and 2 will be assessed in
  terms of level of agreement with these derived
  responses.

11
Receiver Operating Characteristic (ROC)
The ROC graph presents a visual summary of the
way a predictive system responds to a sample of
real cases, some of which will be on the
borderline. A perfect predictive system would
follow the left and upper boundaries of the
graph, one which is no better than chance would
follow the leading diagonal.
12
Chi-squared test (1)

• We have gathered data on TP, TN, FP, FN responses
  of 2 groups.
• Are differences between groups due to systematic
  factors or random variation?
• Standard chi-squared test
• start from contingency table
• calculate chi-squared figure
• compare to characteristic value for desired
  degree of certainty and size of contingency table
• testing for null hypothesis (absence of
  systematic difference)

13
Chi-squared test (2)
14
Experimental Conclusion (1)

• No significant difference in the performance of
  the code review exercise was found between the
  control group and the treatment group. We
  therefore conclude that there is no evidence from
  the current experiment to suggest that the
  metrics information is of benefit in the setting
  simulated by this experiment.

15
Experimental Conclusion (2)

• While the experiment failed to demonstrate a
  significant difference between the performance of
  the two groups, nonetheless it is possible to say
  that a difference in performance was observed.
  Although the statistical analysis of the data
  shows that the difference observed was well
  within the range of outcomes that might arise out
  of the operation of random effects, it is
  possible that a similar experiment with a larger
  number of participants might demonstrate a
  significant effect.

16
Project Summary

• Nature of source code metrics
• Selection of existing and new metrics based on
  GQM paradigm
• Evidence on industry attitudes
• Empirical evaluation of tools
• human decision support context
• problems of realistic experimentation
• burden of proof
• appropriate statistical techniques