MetriCon 2.0

1
MetriCon 2.0

• Correlating Automated Static Analysis Alert
  Density to Reported Vulnerabilities in Sendmail
• Michael Gegick, Laurie Williams
• North Carolina State University
• 7 August 2007

2
Introducing Security Parallels
Component any logical part of the software
system 1
Reliability context (well-established)
Security context (new)

• Fault-prone component
• Likely to contain faults
• Failure-prone component
• Likely to have failures in field

Vulnerability-prone component Likely to contain
vulnerabilities Attack-prone component Likely
to be exploited in the field
Make informed risk management decisions and
prioritize redesign, inspection, and testing
efforts on components.
1 IEEE, "ANSI/IEEE Standard Glossary of
Software Engineering Terminology (IEEE Std
610.12-1990)," Los Alamitos, CA IEEE Computer
Society Press, 1990.
3
Early Reliability Metrics

• Static analysis
• N. Nagappan and T. Ball, "Static Analysis Tools
  as Early Indicators of Pre-release Defect
  Density," in International Conference on Software
  Engineering, St. Louis, MO, 2005, pp. 580-586.
• J. Zheng, L. Williams, W. Snipes, N. Nagappan, J.
  Hudepohl, and M. Vouk, "On the Value of Static
  Analysis Tools for Fault Detection," IEEE
  Transactions on Software Engineering, vol. 32,
  pp. 240-253, 2006.
• Complexity metrics
• J. Munson and T. Khoshgoftaar, "The Detection of
  Fault-Prone Programs," IEEE Transactions on
  Software Engineering, vol. 18, pp. 423-433, 1992.
• T. Khoshgoftaar and J. Munson, "Predicting
  Software Development Errors using Software
  Complexity Metrics," IEEE Journal on Selected
  Areas in Communications, vol. 8, pp. 253-261,
  1990.
• Historical (failure)
• N. Nagappan, T. Ball, and A. Zeller, "Mining
  metrics to predict component failures," in
  International Conference on Software Engineering,
  Shanghai, China, 2006.
• T. J. Ostrand, E. J. Weyuker, and R. M. Bell,
  "Where the bugs are," in International Symposium
  on Software Testing and Analysis, Boston,
  Massachusetts, 2004, pp. 86-96
• Object-Oriented metrics
• V. Basili, L. Briand, and W. Melo, "A Validation
  of Object Oriented Design Metrics as Quality
  Indicators," IEEE Transactions on Software
  Engineering, vol. 21, 1996.
• Y. Zhou and L. Hareton, "Empirical Analysis of
  Object-Oriented Design Metrics for Predicting
  High and Low Severity Faults," IEEE Transactions
  on Software Engineering, vol. 32, no. 10, 2006,
  pp. 771-789.

4
Research Objective

• Build and validate models for predicting
  vulnerability- and attack-prone components based
  upon security-based automated static analyzer
  (ASA) alerts
• Metric ASA alert density and severity early in
  the development phase
• ASA cannot find all types of security
  vulnerabilities
• Are ASA alerts a good predictor?
• Implementation bugs, design flaws, operational
  vulnerabilities
• Software engineers plug the number of security
  alerts into the predictive models to determine
  which components are vulnerability- and
  attack-prone.

5
Building the Initial Predictive Model

• Generalized linear model (data are not normally
  distributed)
• Poisson distribution?
• mean number vulnerabilities in component
• estimated intercept
• estimated slope
• value of random variable alert density of
  component

6
Feasibility Study

• Fortify Softwares Source Code Analyzer (SCA)
• Scanned ten releases of Sendmail
• 8.12.2-8.12.11
• 996 total files scanned
• 21 potential vulnerabilities
• Vulnerabilities reported in RELEASE_NOTES
• Nine vulnerabilities with known exploits

7
Feasibility Study vulnerability-prone

• Poisson distribution
• Models the response data
• Reported vulnerability
• Association between Hot alert density and number
  of vulnerabilities per reported per file
• Positive slope ? positive association between
  alerts and reported vulnerabilities
• p-value ? high significance in association
• Standard error ? substantial overdispersion
• Few data points

Slope p-value Chi-Square /df Goodness- of- fit measure Standard error
294.8069 0.0016 1.1939 93.3422
8
Feasibility Study attack-prone

• Poisson distribution
• Models the response data
• Number of known exploits (nine) for a Sendmail
  file
• Association between Hot alert density and number
  of known exploits
• Slope ? positive association between alerts and
  exploits
• p-value ? low significance
• Standard error ? substantial overdispersion
• Few data points

Slope p-value Chi-Square /df Goodness- of- fit measure Standard error
140.4334 0.4980 1.2099 207.2419
9
Questions

• Thank you!
• mcgegick_at_ncsu.edu