Software%20Assurance%20Metrics%20and%20Tool%20Evaluation%20(SAMATE)

1
Software Assurance Metrics and Tool Evaluation
(SAMATE)

• Michael Kass
• National Institute of Standards and Technology
• http//samate.nist.gov/
• Michael.kass_at_nist.gov

2
Outline

• Overview of Software Assurance (SwA) tool testing
  at NIST
• Description of SAMATE project
• Follow-on

3
Dept Homeland Security Concern

• Do software assurance tools work as they should?
• Do they really find vulnerabilities and catch
  bugs? How much assurance does running the tool
  provide?
• Software Assurance tools should be
• Tested (accurate and reliable)
• Peer reviewed
• Generally accepted

4
Goals of SAMATE

• Develop metrics for the effectiveness of SwA
  tools and to identify deficiencies in software
  assurance methods and tools
• Perform SwA RD to assess current methods and
  tools in order to identify deficiencies which can
  lead to software product failures and
  vulnerabilities
• Identify gaps in methods and tools and suggest
  areas of research

5
The NIST SAMATE Project(Software Assurance
Metrics and Tool Evaluation)

• Conduct surveys
• Tools
• Researchers and companies
• Host workshops conference sessions
• Taxonomy of SwA functions and techniques
• Order of importance (cost/benefit, criticalities,
  )
• Gaps and research agendas
• Studies to develop tool effectiveness metrics
• Evaluate tools
• Detailed specification
• Test plans
• Host reference dataset library

6
A Taxonomy of Static Analysis Tool Functions

• Language
• Source/Binary analysis
• Semantic checking (abstract syntax tree)
• Interprocedural analysis
• Strong type checking (type casting
  vulnerabilities, uninitialized variable use)
• Memory allocation checking (memory leaks,
  deallocation of unallocated memory)
• Logic checking (unnecessary code, unreachable
  code)
• Interface checking (include file cycling)
• Security checking
• Buffer overflow/underflow
• Stack overflows
• Heap overflows
• Integer overflow/underflow
• Tainted data
• Error path problems
• Locking problems
• Code metric generation (LOC, number of methods,
  levels of inheritance)

7
SA Tool Effectiveness Metrics

• What constitutes a tools effectiveness metric?
• Number of defects detected vs. total defects
• Number of false positives
• Number of false negatives

8
Documenting tool effectiveness

• Tool functional specification
• Test plan
• Reference dataset
• Test report

9
SAMATE Project Timeline
T(mos.)
1
2
6
12
18
24
3
4
5
9
15
21
Tool Survey
Function Taxonomy
Survey Publication
tool testing matrix
Spec0
test reports
select func
test plan
Spec1
strawman spec
test plan
draft
test reports
select func
Spec0
test reports
test plan
Spec1
strawman spec
test plan
test reports
draft
10
Contact for SAMATE Participation

• Paul Black
• Project Leader, Software Diagnostics
  Conformance Testing Division, Software Quality
  Group
• paul.black_at_nist.gov