Improve Software Quality with Fault Injection

1
Improve Software Quality with Fault Injection

• Rob Grzywinski
• rgrzywinski_at_yahoo.com

2
Speaker Introduction

• Started writing software interfaces for
  scientific equipment
• Managed a successful consulting company
• Exposure to projects and products running from
  content management / publishing to security
• Forced a hard look at effective software
  maintenance
• Technology due diligence for Silicon Valley VCs
• Vastly increased exposure to software
  technologies
• Currently working as a startup manager
• A Business-Centric Approach to Information
  Security
• Focuses on risk management

3
Motivation

• Writing and testing an IO-intensive application
• Robust, asynchronous event (message) collector
• NIO (early 1.4 days)
• How to best simulate and test various failure
  modes?
• NIO buggy as all get-out
• Networks are inherently flaky and problematic
• Didnt want to have to obfuscate or drastically
  change code in order to test it

4
Example

• void readFile() throws IOException
• ...
• final InputStream is new FileInputStream()
• ...
• while((offset lt bytes.length)
• (numRead is.read(bytes, offset,
  (bytes.length - offset))) gt 0)
• offset numRead
• ...
• (from http//javaalmanac.com/egs/java.io/File2Byte
  Array.html)

• What are the things that could go wrong with this
  code?
• new FileInputStream() can throw
  FileNotFoundException
• InputStream.read() can throw IOException and
  IndexOutOfBoundsException and can return -1 for
  end of file

How do we test how the application responds to
one of these situations? Specifically, how do we
cause these situations to happen so that we can
test how the application responds?
5
Possible Techniques

• Force the situations at the OS level
• Quite hairy to reproduce reliably when theyre
  needed
• Refactor the snot out of it
• Replace the call to InputStream.read() to some
  local instrumented method
• Create our own instrumented InputStream subclass
  possibly using mock objects
• Inject the subclass via IoC (requires some
  framework such as PicoContainer or Spring)
• Just comment out the code and replace with throw
  new IOException()
• Egad!
• We need a way to inject a fault without changing
  the code!

6
Agenda

• Quick Introduction to Fault Injection
• Why Use Fault Injection
• Fault Injection Examples
• Beyond Fault Injection
• Quick Introduction to AOP
• Fault Injection in Java
• AspectJ
• Javassist
• Wrap Up

7
Fault Injection 101

• A fault is an abnormal condition or defect which
  may lead to a failure 10
• Fault injection involves the deliberate insertion
  of faults or errors into a computer system in
  order to determine its response 9

8
Fault Injection 101 (cont)

• Traditionally, Fault Injection is specific to
  faults derived from hardware
• Disk faults such as file missing, file not
  writable, and corrupt file
• Network faults such as host not found, host
  inaccessible, and high latency
• Memory faults such as corrupt memory
• Operating system faults such as interrupts or
  know bugs
• Traditionally used in mission-critical fault
  tolerant environments

9
Fault Injection 101 (cont)

• Fault Injection comes in two favors
• Hardware-based
• Typically requires specialized hardware
• Software-based
• Traditionally attempts to mimic hardware-based
  fault injection and typically involves direct
  interaction with the operating system
• More recently expanded to include any fault that
  can occur in software such as
• Mutated bytecode
• Exceptions
• Invalid or mutated input / output values
• Deadlock / resource contention
• Application Security
• etc.

10
Fault Injection 101 (cont)

• There are two primary steps to Fault Injection
• To identify the set of faults that can occur
  within an application, module, class, method,
  etc.
• e.g. if the application does not use the network
  then theres no point in injecting network faults
• In practice this isnt as easy as it sounds
• To exercise those faults to evaluate how the
  application responds
• Does the application detect the fault, is it
  isolated and does the application recover from
  it? 8

11
Why Use Fault Injection

• Fault occurred in a production application and it
  is determined that that is a valid fault (i.e. it
  is expected to occur)
• Inject the fault in a testing environment without
  having to reproduce the conditions that actually
  caused the fault
• Any fault that is difficult or inconvenient to
  reproduce in testing
• Out of memory
• Disk full
• Database crash
• 3rd party library / Java bug
• Deadlock
• Resource contention
• Dumb users
• Invalid user
• Corner cases

?
12
Why Use Fault Injection (cont)

• You publish an API and you want to be tolerant of
  user input or response
• It is inconvenient or undesirable to refactor
  code to facilitate testing
• e.g. 3rd party library
• You believe that testing code coverage is a good
  metric to measure for code quality 11
• Injecting faults forces exception paths to be
  followed
• More to come

13
Reminder

• You dont have to recreate the conditions that
  caused the fault
• you only have to recreate the fault
• e.g. if an OhNoException crashed your application
  because some little old lady drove her 1966 Buick
  Skylark into your datacenter, you dont need the
  Skylark or the little old lady to fix the
  application

14
Examples

• Reading from a file

final InputStream is new FileInputStream() ..
. while((offset lt bytes.length) (numRead
is.read(bytes, offset, (bytes.length -
offset))) gt 0) offset numRead ...

• Faults for InputStream.read()
• The file is corrupt
• The file is removed / truncated / appended while
  reading
• The user does not have permission to read the file

15
Examples (cont)

• Creating a directory and temporary file

final File directory new File() final
boolean success directory.mkdirs() if(success)
final File file File.createTempFile(, ,
directory) ...

• Faults
• Making the parent directory(s) fails
• Temp file creation failure
• Permissions
• Parent directory(s) deleted
• Injection of a specific temp file

16
Examples (cont)

• Cant easily show in examples
• Deeply nested exceptions
• Effects of unchecked exceptions
• Consequences of using catch(Exception e)
• Complex / real-world cases

17
Intelligent Fault Injection

• Fault injection doesnt have to be all on or all
  off
• Logic can be coded around injected faults
• e.g. InputStream.read()
• Throw IOException after n bytes are read
• Return -1 (EOF) one byte before the actual EOF
• Sporadically mutate the read bytes

18
Beyond Fault Injection

• The techniques that Im going to demonstrate
  arent specific to injecting faults (i.e.
  abnormal conditions or defects)
• What if analysis
• Test difficult corner cases
• The techniques are similar to mock objects or IoC
  but much more granular

19
Summary

• Intercept operations and inject code to
• Perform a custom function
• Return a custom value
• Modify a input parameter
• Throw an exception
• Set the state of an object to a specific value

20
Checkpoint

• Done
• Quick Introduction to Fault Injection
• Why Use Fault Injection
• Fault Injection Examples
• Beyond Fault Injection
• Next
• Quick Introduction to AOP
• Fault Injection in Java
• AspectJ
• Javassist
• Wrap Up

21
Checkpoint (cont)

• Questions?

22
Aspect Oriented Programming 101

• Join point
• a point in the flow of a running program 14
• It is the where of
• Not all AO languages support all join points.
  e.g. AspectJs join points are
• Well-defined points in the execution of a
  program. Not every execution point is a join
  point only those points that can be used in a
  disciplined and principled manner are. So, in
  AspectJ, the execution of a method call is a join
  point, but "the execution of the expression at
  line 37 in file Foo.java" is not. 2

23
Aspect Oriented Programming 101 (cont)

• Join point examples
• reading or writing a field
• calling or executing a method or constructor
• catching or throwing an exception
• A join point shadow is the location of a join
  point in the source code or bytecode of the
  program 5
• We will use join point and join point shadow
  interchangeably

24
Aspect Oriented Programming 101 (cont)

• Pointcut
• a set of join points 6
• Basically a query where the join points are the
  data that is being queried

25
Aspect Oriented Programming 101 (cont)

• Advice
• code that executes at each join point picked
  out by a pointcut 7
• Advice is the code that youre injecting

26
Economy of (your country)

• Explain which goods and services are produced in
  your country. How do people typically provide for
  the needs of themselves and their families?

27
Economy of (your country)

• Explain which goods and services are produced in
  your country. How do people typically provide for
  the needs of themselves and their families?

28
Economy of (your country)

• Explain which goods and services are produced in
  your country. How do people typically provide for
  the needs of themselves and their families?

29
Economy of (your country)

• Explain which goods and services are produced in
  your country. How do people typically provide for
  the needs of themselves and their families?

30
Economy of (your country)

• Explain which goods and services are produced in
  your country. How do people typically provide for
  the needs of themselves and their families?

31
Tourism in (your country)

• Tell about the points of interest in your country
  that people from other countries may be
  interested in visiting.