Techniques for Validating the Security Quality of Infrastructure Software

1
Techniques for Validating the Security Quality of
Infrastructure Software

• John D. McGregor
• johnmc_at_cs.clemson.edu

2
Outline

• Motivation
• Proposed strategy
• Detailed actions
• Conclusion

3
Theme Life Cycle Threats

• There is not a means for automated testing of
  large software, both static and mobile code, to
  detect, identify malicious code, sleeper codes,
  and exploitable vulnerabilities and to determine
  and understand the potential impact on the
  life-cycle of the codes. Current testing
  approaches are largely manual rather than
  automated.
• CSIIR Workshop Themes document

4
A Recommended Strategy

• Software security vulnerabilities are often
  caused by defective specification, design, and
  implementation.
• require that software be designed with security
  at the very heart of the design process
• Establish a security verification and validation
  program to evaluate different software
  development processes and practices for
  effectiveness in producing secure software.
• Certify those processes demonstrated to be
  effective for producing secure software.
• Security Across the Software Development
  Lifecycle Task Force

5
Recommended Practices

• Statistical testing - Usage based testing permits
  valid statistical estimation of quality with
  respect to all the executions not tested and
  tends to find any remaining high-failure-rate
  defects early.
• Production testing - Two strategies are testing
  security functionality with standard functional
  testing techniques, and risk-based security
  testing based on attack patterns and threat
  models. A good security test plan (with
  traceability back to requirements) uses both
  strategies.
• Process models - organizations can use the goals
  and attributes defined in process models as
  high-level guides for defining and improving
  their management and engineering processes in the
  ways they feel are most appropriate for them.
• Security Across the Software Development
  Lifecycle Task Force

6
SEI

• The security of a software-intensive system is
  directly related to the quality of its software1.
  
• Over 90 of software security incidents are
  caused by attackers exploiting known software
  defects.
• Analysis of 45 e-business applications showed
  that 70 of security defects were design defects.
  
• Experienced and capable software engineers
  inject, on average, one defect every nine lines
  of code.
• A one million line of code systems typically
  contains 1,000-5,000 defects when shipped.
• 1 http//www.sei.cmu.edu/tsp/tsp-security.html

7
A Final Source

• One of the key things that developers can do to
  help secure their systems is to write code that
  can withstand attack and use security features
  properly.
• Defend Your Code with Top Ten Security Tips Every
  Developer Must Know
• 8 out of 10 tips are directly programming issues
• http//msdn.microsoft.com/security/securecode/defa
  ult.aspx?pull/msdnmag/issues/02/09/securitytips/d
  efault.aspx

8
Basic Assumptions

• Testing early is cheaper than testing later.
• System test of a system with j modules and k
  experiments (iterations or different
  implementations for a given interface) results in
  the following cost of testing

• Where ct is the cost per test.
• Module-level tests with j modules and k
  experiments results in

Baldwin and Clark. Design Rules The Power of
Modularity, volume 1, MIT Press, 2000.
9
Basic assumptions - 2
Though estimates vary, the cost of removing
defect increases dramatically later in the life
cycle.
Design Design Inspection Code Development Test Acceptance Production
IBM 1 1.5 60 100
Boehm 1 15 to 40 30 to 70 40 to 1000
Remus 1 20 82
Ackerman 1 2 to 10
Russell 1 2 to 4 30
10
Premise

• Our premise is that poorly written software will
  contain more vulnerabilities than well written
  software where the security quality attribute is
  a design driver.
• Current views of security often take a defensive
  approach. Some of the security infrastructure
  even adds to the security risk due to the
  complexity it adds to the product.
• We propose an offensive approach in which
  security is a key design driver and a priority
  throughout the development process.

11
Context

• A chain of quality should be threaded through the
  entire process so that validation is most
  effective and efficient.

12
A Proposed Strategy

• Develop method engineering tactics and guidelines
  that enhance the security quality of the software
  through improved processes.
• Structure architecture evaluation techniques to
  focus on security by searching for static
  security patterns.
• Discover and capture test patterns that
  correspond to dynamic security patterns.
• Develop focused test techniques to effectively
  explore security test patterns while reducing the
  test suite size.
• Create a defect model for security that can be
  used to predict types and number of security
  vulnerabilities in scientific codes.

13
Action Develop method engineering techniques

• Method engineers create custom-made processes to
  help a project achieve specific goals.
• The goal of being secure needs to be
  operationalized so that these engineers can
  assemble methods in ways that enhance the
  security of the software product built using
  their processes.
• This task would involve extending the Software
  Process Engineering MetaModel1 (SPEM) standard to
  define security-specific constructs. The model
  would be automated using existing tools such as
  MetaEdit.
• 1 Software Process Engineering Metamodel, version
  1.1, Object Management Group (OMG), 2005.

14
Action Develop method engineering techniques - 2

• The security-oriented method fragments would be
  suitable to integrate into processes defined in
  the context of the SPEM.
• Process audits could evaluate the strength of the
  security aspect of the process, once it is
  explicitly embedded in the process, just as other
  qualities are validated.
• Deliverables A process definition guide that
  would show how to design security-centric
  development process fragments, an assembly guide,
  and an example process.

15
Action Create architecture security analysis

• Architects have techniques that they apply to an
  architecture in order to improve the behavior of
  the architecture with respect to classes of
  security threats1.
• Assuring system survivability requires showing
  that the system architecture is adequately
  resilient to likely patterns of attack2.
• One approach to architecture design is to
  identify quality attributes for the architecture
  and make design decisions that enhance the
  desirable qualities and degrade the less
  desirable ones.
• Architecture evaluation techniques such as the
  architecture trade-off analysis method (ATAM) can
  be used to focus architecture evaluations on the
  security quality through security-specific
  scenarios.
• One aspect of this task would be to develop a set
  of architecture-level security scenarios that
  would guide the evaluation of an architecture for
  the security quality.
• 1 Security and Survivability Reasoning Frameworks
  and Architectural Design Tactics.
  CMU/SEI-2004-TN-022
• 2 Architectural Refinement for the Design of
  Survivable Systems CMU/SEI-2001-TN-008

16
Action Create architecture security analysis - 2

• This evaluation can be automated if the
  architects use a formal architecture description
  language (ADL).
• Our current implementation uses the Architectural
  Analysis and Design Language (AADL) from the
  Society of Automotive Engineers (SAE) and Eclipse
  plug-ins.
• A second part of the task could be to identify
  standard mechanisms for representing security in
  an architecture description so that security
  patterns could be automatically recognized.
• Deliverables A set of validation scenarios and
  specific architecture tactics for recognizing
  security vulnerabilities at the architecture
  level

17
Current implementation as Eclipse plug-in for AADL
18
Action Create architecture security analysis - 3

• This analysis could also be conducted on
  pre/post-test basis
• Use an architecture re-construction technique
  such as OAR1 to extract the as built
  architecture from the implementation.
• Compare to the as designed architecture to
  determine if changes have occurred
• 1 Options Analysis for Reengineering (OAR) A
  Method for Mining Legacy Assets
  CMU/SEI-2001-TN-013

19
Action Discover test patterns for security

• Test patterns are solutions to testing problems
  in context. They often correspond to design
  patterns.
• Test patterns are language independent
• Content of a test pattern1
• Problem - Description of pattern to be tested
• Context - Special testing conditions
• Forces - What types of faults are we looking for?
• Solution - Test case selection strategy that
  tests the interactions among the components that
  implement the pattern
• Example A sample implementation
• 1 John McGregor and David Sykes Practical Guide
  to Testing Object-Oriented Software,
  Addison-Wesley, 2001.

20
Action Discover test patterns for security - 2

• Problem The synchronous communication between
  two objects is modified to be asynchronous by
  adding a callback object. How should this be
  tested?

Client
Client
Callback
Server
Server
21
Action Discover test patterns for security - 3

• Context
• An intermediate object forwards messages
• Forces
• Possible to intermingle successive
  messages/responses
• Temporal considerations must be added to the test
  case
• Solution
• Construct tests that exercise the callback in a
  variety of states
• Construct multiple clients, submit multiple
  requests through multiple callback objects
• Build test cases that submit a second message
  prior to receiving the response from the first
  message

22
Action Discover test patterns for security - 4

• In this task, security vulnerabilities and
  standard designs to correct them would be
  studied.
• The standard designs would lead to a catalog of
  specific test patterns.
• Users of the test patterns could implement them
  in whatever language was in use.
• Deliverables test pattern catalog

23
Task force identified security patterns
49 security patterns have been identified. Here
are a few.
1. Make the Client Invisible 2. Target Programs
That Write to Privileged OS Resources 3. Use a
User-Supplied Configuration File to Run Commands
That Elevate Privilege 4. Make Use of
Configuration File Search Paths 5. Direct Access
to Executable Files 6. Embedding Scripts within
Scripts 7. Leverage Executable Code in
Non-executable Files 8. Argument Injection 9.
Command Delimiters 10. Multiple Parsers and
Double Escapes
11. User-Supplied Variable Passed to File System
Calls 12. Postfix NULL Terminator 13. Postfix,
Null Terminate, and Backslash 14. Relative Path
Traversal 15. Client-Controlled Environment
Variables 16. User-Supplied Global Variables
(DEBUG1, PHP Globals, and So Forth) 17. Session
ID, Resource ID, and Blind Trust 18. Analog
In-Band Switching Signals (aka Blue Boxing) 19.
Attack Pattern Fragment Manipulating Terminal
Devices 20. Simple Script Injection
Security Across the Software Development
Lifecycle Task Force
24
Action Develop focused test techniques - 1

• Operational profiles establish a relative
  frequency of operations for a particular type of
  user or application.
• Test cases are chosen with the same relative
  frequency to mimic actual use or to mimic attack
  scenarios.
• This narrows the range of values to cover in test.

User Operation Relative frequency
User1 Op1 .2
User1 Op2 .6
User1 Op3 .2
User2 Op1 .4
User2 Op2 .3
User2 Op3 .3
25
Action Develop focused test techniques - 2

• Testing all possible values even once, much less
  all possible combinations of values, is usually
  impossible.
• Combinatorial test designs can systematically
  sample test values to ensure the maximum coverage
  with minimum test cases.
• Orthogonal array testing is one combinatorial
  approach.

26
OATS tool - 1

• Three variables, each with three possible values,
  would require 27 test cases for all combinations
  coverage.

Database driver JDBC1.0 JDBC2.0 ODBC1.0
browser Netscape IE5.0 Firefox
of web servers 5 50 200
27
OATS tool - 2

• OATS reduces this to 9 test cases, testing
  pair-wise but remains approximately 96 as
  effective as all combinations coverage at
  finding defects.

1 JDBC1.0 Netscape 200
2 JDBC1.0 IE5.0 50
3 JDBC1.0 FoxFire 5
4 JDBC2.0 Netscape 50
5 JDBC2.0 IE5.0 5
6 JDBC2.0 FoxFire 200
7 ODBC1.0 Netscape 5
8 ODBC1.0 IE5.0 200
9 ODBC1.0 FoxFire 50
28
Action Develop focused test techniques - 3

• Pair-wise value combinations can reduce the size
  of the test suite dramatically
• These techniques have been applied in domains
  such as telecommunications.
• In this task we would develop an industrial
  strength tool that allows the design of
  combinatorial test suites.

29
Action Develop focused test techniques - 4

• This technique is used in conjunction with
  traditional functional and structural strategies
  for test case selection.
• Functional tests ensure that all specified
  functions are correct and structural tests ensure
  that all feasible paths are valid.
• The result is confidence that the software does
  everything it is supposed to and nothing that it
  Is not supposed to.
• Combinatorial techniques increase the coverage of
  large software products without significant
  increases in test effort.
• This approach allows for risk-based test
  selections.
• Deliverables software tool that uses
  combinatorial techniques and risk-based
  techniques to define effective test cases.

30
Action Develop a security defect model for
scientific codes

• If you just found the 73rd defect in your 50,000
  LOC program, do you feel good about it?
• With a validated defect model you would at least
  know how to feel.
• You would know approximately how many defects to
  expect and what types of defects to look for.

31
Action Develop a security defect model for
scientific codes - 2

• Using the relative frequency and estimated size,
  estimates can be made of the number of defects
  and the test case selection process be more
  precisely directed.

Security defect type Relative frequency per KLOC
Static queue sizes 60
HTTP cookies 30
Argument injection 10

Total 100
32
Action Develop a security defect model for
scientific codes - 3

• In this task we would develop a defect model
  framework for scientific codes using historic
  data.
• The framework would be packaged with a
  specialization method for tailoring the framework
  to a specific development context.
• This would allow us to provide statistical
  estimates of the number of defects remaining in a
  software product and to provide a confidence
  interval on statistics such as reliability.
• Deliverables defect model

33
Conclusion

• No single action or algorithm will produce secure
  software. There must be a chain of quality
  activities.
• The outlined strategy places security quality
  gates at several places in the software product
  development life cycle.

34
Conclusion - 2

• Implementing any part of this strategy will
  improve the security of the software products
  produced.
• Develop method engineering tactics and guidelines
  that enhance the security quality of the software
  through improved processes.
• Structure architecture evaluation techniques to
  focus on security by searching for static
  security patterns.
• Discover and capture test patterns that
  correspond to dynamic security patterns.
• Instrument combinatorial test techniques to
  effectively explore security test patterns while
  reducing the test suite size.
• Create a defect model for security that can be
  used to predict types and number of security
  vulnerabilities in a given product.