Milk or Wine: Does Software Security Improve with Age?

1
Milk or WineDoes Software Security Improve with
Age?
Andy OzmentMIT Lincoln Laboratory
Stuart SchechterMIT Lincoln Laboratory

• 15th Usenix Security Symposium
• 3 August 2006

This work is sponsored by the I3P under Air Force
Contract FA8721-05-C-0002. Opinions,
interpretations, conclusions and recommendations
are those of the author(s) and are not
necessarily endorsed by the United States
Government. This work was produced under the
auspices of the Institute for Information
Infrastructure Protection (I3P) research program.
The I3P is managed by Dartmouth College, and
supported under Award number 2003-TK-TX-0003 from
the U.S. Department of Homeland Security, Science
and Technology Directorate. Points of view in
this document are those of the authors and do not
necessarily represent the official position of
the U.S. Department of Homeland Security, the
Science and Technology Directorate, the I3P, or
Dartmouth College.
2
or

• Does Software Security Improve with Age?

Andy OzmentUniversity of Cambridge MIT
Lincoln Laboratory
Stuart SchechterMIT Lincoln Laboratory
3
Are rates of vulnerability reports decreasing

• Time

4
or not?

• Time

5
The OpenBSD data set140 vulnerabilities
reported in 7.5 years

• An event is a vulnerability if
• It is listed in the OpenBSD security
  page, Bugtraq, OSVDB, NVD (ICAT), or ISS
  X-Force
• There is no vulnerability of the same type
  reported that week

Vulnerabilities introduced ()
?
Version
6
Study period
3.7
2.3
NetBSD
7
Vulnerability lifecycle
8
Vulnerabilities time of death
Vulnerability deaths
Version current
9
Vulnerabilities time of death
Birth version
Vulnerability deaths
?
Version current
10
Why so many foundational vulnerabilities?
?
?
11
Source code composition over time
12
Determining a versions code contribution
We analyzed all C code files and compared each
versions (We ignored code specific to platforms
other than x86)
.ch
v2.7
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do
look_at_speaker() write_notes() while
(clock() lt end)
/ Used while attending conferences
/ void attend_talk(int end) sit() do
check_email() compose_email()
look_at_speaker() while (clock() lt end)
13
Determining a versions code contribution
1. We removed all comments.
.ch
v2.7
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do
look_at_speaker() write_notes() while
(clock() lt end)
/ Used while attending conferences
/ void attend_talk(int end) sit() do
check_email() compose_email()
look_at_speaker() while (clock() lt end)
14
Determining a versions code contribution
2. We removed/ignored all white space.
.ch
v2.7
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do look_at_
speaker() write_notes() while(clock()ltend)
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while(c
lock()ltend)
15
Determining a versions code contribution
3. We counted the lines in the more recent
version.
.ch
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while
(clock() lt end)
1 2 3 4 5 6 7 8
16
Determining a versions code contribution
3. A diff was performed to identify lines of code
derived entirely from the earlier version.
.ch
v2.7
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while
(clock() lt end)
/ Used while attending conferences
/ void attend_talk(int end) sit() do look_at_
speaker() write_notes() while(clock()ltend)
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while
(clock() lt end)
17
Determining a versions code contribution
4. We counted the lines contributed by the
earlier version.
.ch
v3.2
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while
(clock() lt end)
1 2 3 4 5 6
v3.2
v2.7
18
Determining all code contributions
5. We repeated the process, overlaying earlier
versions.
.ch
v3.2
v2.6
example.c
/ Used while attending conferences
/ void attend_talk(int end) sit() do check_ema
il() compose_email() look_at_speaker() while
(clock() lt end)
1 2 3 4 5
v3.2
v2.7
v2.6
look_at_speaker()
19
Source code composition over time
Somethings fishy here!
20
Source code composition over time
Lines of code contributed by each earlier version
Contributor version
Composite version
21
Complications in source comparison
cp xwin/compress/zip.c batman/
compress/
lz.h arith.c shrink.c maketiny.c zip.c
bang.h bang.c wham.c splat.c
zip.c
22
Complications in source comparison
v2.3
v2.4
compress/
compress/
lz.h arith.c shrink.c maketiny.c zip.c
zip.c
lz.h arith.c shrink.c maketiny.c zip.c
bang.h bang.c wham.c splat.c
zip.c
?
/zip.c
contents(v2.3/src/xwin/zip.c)?contents(v2.4/src/t
ools/batman/zip.c)
23
Complications in source comparison
v2.3
v2.5
compress/
lz.h arith.c shrink.c maketiny.c zip.c
zip.c
24
Source code composition over time
artifact
xfree86 v3?v4
artifact
25
What was the question, again?

• Does software security
• improve with age?

26
Data set, revisited
Vulnerabilities introduced ()
Version
27
A more precise question

• Does software security
• improve with age?

Is the rate of vulnerability reporting in OpenBSD
2.3 code decreasing over time?
28
Raw data time between vulnerability reports
Mean 29.1 Median 18 ? 29.1 Range 1 - 126
29
Eight slices of raw data
Mean 29.1 Median 18 ? 29.1 Range 1 - 126
30
Eight slices of raw data
Mean 29.1 Median 18 ? 29.1 Range 1 - 126
31
Number of vulnerabilities per study eighth

• Confidence intervals derived froma normal
  approximation of a homogenous Poisson process

32
Two slices of raw data
Mean 29.1 Median 18 ? 29.1 Range 1 - 126
33
Number of vulnerabilities per study half
Confidence intervals derived from a normal
approximation of a homogenous Poisson process
34
Previous literature

• Eric Rescorla Is Finding Security Holes a Good
  Idea?
• 2004 Workshop on Economics and Information
  Security (WEIS)
• Argued vulnerability reporting rate was not
  decreasing
• 1. Used Laplace test
• 2. Attempted to fit reliability growth models
• Result affected by shortcomings inherent to NVD
  db
• May not list all affected versions
• Death date is based on date public
• Death date noisy
• Only vulnerabilities with CVE labels

35
Laplace test
H1 Increasing rate
95
90
Assumes vulnerability reporting is a
non-homogenous Poisson process
H0 No trend
90
95
H2 Decreasing rate
Age of vulnerability (years)
36
Laplace test results
H1 Increasing rate
H0 No trend
H2 Decreasing rate
37
Reliability growth models

• Tested 8 models
• Musas Logarithmic model fit
• Acceptable one-step-aheadpredictive accuracy
• Intensity
• Start 0.051 vulnerability/day (19.6
  days/vulnerability)
• End 0.024 vulnerability/day (41.7
  days/vulnerability)
• Estimated number ofvulnerabilities remaining
• 42

38
Wine!

• Code security of OpenBSD 2.3 is improving with age

39
Limitations

• Cannot normalize for changes in
• Number of vulnerability hunters
• Effort expended by vulnerability hunters
• Vulnerability hunting fads
• Image file processing
• Microsoft
• Data set represents vulnerability hunting
  environmentduring the course of the study

40
A few more juicy tidbits

• But while were here
• Correlation LOC and number of vulnerabilities?
• What are vulnerability densities like?
• What is the median lifetime of a 2.3
  vulnerability?

41
Lines of code vs. number vulnerabilities
No correlation
42
Vulnerability densities
Vers. vulns 1M LOC vulns1M LOC
2.3 87 10.14 8.58
2.4 14 0.42 33.04
2.5 6 0.27 21.84
2.6 9 1.05 8.55
2.7 7 0.77 9.10
2.8 0 0.40 0.00
2.9 4 2.23 1.79
3.0 5 0.62 8.00
3.1 2 0.81 2.46
3.2 2 0.33 6.03
3.3 2 0.32 6.18
3.4 0 0.83 0.00
3.5 2 1.44 1.39
3.6 0 0.74 0.00
3.7 0 0.91 0.00
Total 140 21.45 6.53

• Wide range of vulnerability densities
• Inflated LOC counts
• Version 2.4
• Internet Key Exchange (2)
• OpenSSL (3)
• Compare with conventional wisdom
• Vulnerabilities lt 0.033 / 1K LOC
• Defects 8-12 / 1K LOC
• But vulnerability counts dont include every
  instance

43
Foundational vulnerabilitiesMedian lifetime
lower bound
44
Conclusion

• Foundation version contributes the majority
  ofcode vulnerabilities
• Rate of reporting is decreasing
• Decrease is slow
• More than half of foundational vulnerabilitiesrem
  ained 2.6 years after its release

45
Questions
46
Days between reports of vulnerabilities