Distributed In Vivo Testing of Software Applications

1
Distributed In Vivo Testing of Software
Applications

• Matt Chu, Christian Murphy, Gail Kaiser
• Columbia University

2
Introduction

• It is infeasible to fully test a large system
  prior to deployment considering
• different runtime environments
• different configuration options
• different patterns of usage
• Extending testing into the deployment environment
  can help reveal faults

3
In Vivo Testing

• We have developed an approach called in vivo
  testing
• Unit tests from the development environment (in
  vitro) are executed at run-time in the deployment
  environment (in vivo)
• Done without affecting the state of the
  application
• However, this approach can incur a large
  performance overhead

4
Distributed In Vivo Testing

• In this work we present a distributed approach
  that takes advantage of networked application
  communities
• Members of the community each run fewer tests
  (thus less overhead) and results are aggregated
  centrally

5
Benefits of Distributed In Vivo Testing

• Amortizing the workload over many instances
  tackles the high performance cost without
  sacrificing the quantity of tests being conducted
  globally
• Having a community of applications
  collaboratively working together increases the
  possibility that an instance will find these
  erroneous permutations of state

6
Overview

• Related work
• In vivo testing the Invite framework
• The distributed Invite framework
• Experimental results
• Conclusions and future work

7
Related Work

• Perpetual testing Clarke SAS00
• Skoll Memon ICSE04
• Gamma Orso ISSTA02
• CBI Liblit PLDI03
• MORABIT Merdes SEM06

8
In Vivo Testing

• Unit tests are executed in the deployment
  environment, in the context of the running
  application, as opposed to a controlled
  development environment
• Tests are run continuously as the application
  runs, at selected points in the program execution
• Tests are run in a separate process (which has
  been created as an exact copy of the original),
  so as not to affect the state of the original
  process

9
In Vivo Testing Defects Targeted

1. Unit tests make incorrect assumptions about the
   state of objects in the application
2. Possible field configurations were not tested in
   the lab
3. A legal user action puts the system in an
   unexpected state
4. An sequence of unanticipated user actions breaks
   the system
5. Defects that only appear intermittently

10
Invite The in vivo testing framework

• Implemented in Java
• Uses AspectJ to weave in the instrumentation at
  compile-time
• Configuration performed by software developer
• Frequency of test execution (per method)
• Reaction policy
• Software deployed as usual, with no requirements
  on end user

11
Invite Running a test
Method foo is about to be executed
Run the test in a separate process
12
Distributed in vivo testing approach

• Amortize the cost of test execution by
  distributing the workload over a number of
  instances
• Cover a variety of different platforms, usage
  patterns, etc.
• Client/Server architecture
• Server can distribute tests according to
  targeting or sampling policies

13
Distributed Invite framework
Invite Server
14
Experiment Setup

• Goal ensure that evenly distributing testing
  load improves performance by decreasing the
  number of tests each individual member in the
  community has to run.
• Subject Jetty WebServer
• 5000 dynamic static web pages requested from
  each instance

15
Experimental Results
overhead
community size
16
Summary

• Overhead declines as the number of members in the
  community grows
• Analytical results
• 18 members 25
• 45 members 10
• 90 members 5
• Optimal minimum overhead (from instrumentation)
  was 2.4

17
Limitations Future Work

• Ensure that the test does not affect the external
  system state
• Improve distribution of tests
• Adjust frequency of test execution based on
  context or resource availability
• Include support for capture/replay, fault
  localization, etc.

18
Conclusion

• Distributed in vivo testing addresses one of the
  biggest problems of the in vivo testing approach
  (extremely high performance penalties) by
  distributing the number of tests that need to be
  run onto the entire community of applications
• The approach maintains the same number of tests
  globally, and also encompasses more diverse
  environments

19
Distributed In Vivo Testing of Software
Applications

• Matt Chu, Christian Murphy, Gail Kaiser
• Columbia University