Automated Systematic Testing of Open Distributed Programs

1
Automated Systematic Testing of Open Distributed
Programs

• By Koushik Sen and Gul Agha
• Presented at FASE 2006

Presented by Jeffrey Votteler for CS527
2
Presentation Overview

• What problem are the authors trying to solve?
• How do the authors propose to solve this
  problem?
• What are the goals of the authors solution?
• What are the results of the authors proposed
  solution?
• Conclusion
• Questions/Comments

3
Problem to be Solved

• Automatic testing of distributed programs
• A program consists of asynchronous processes
  which communicate through asynchronous message
  passing
• Large number of possible data inputs
• Large number of potential behaviors/executions
  due to asynchrony
• Goal of the solution to the problem
• Execute all reachable statements in the
  distributed program
• Detect if there are deadlocks in the program

4
Importance Of Finding Solution

• Distributed programs are difficult to test due to
  their asynchronous behavior
• Importance of finding deadlocks
• Code coverage of all statements
• Concrete examples of bugs/errors to help in
  debugging
• Automatic generation of test cases removes the
  human aspect of testing

5
Proposed Solution / Overview

• Use of Concolic Execution
• Explore all possible and distinct behaviors of
  the distributed program
• Use symbolic inputs in place of concrete values
  to determine alternative behaviors at constraints
• Use concrete data to determine the partial order
  of events in an execution (concrete path)
• Avoid equivalent tests/executions
• Each conditional in program is a constraint that
  determines execution path

6
Proposed Solution / Overview

• Simultaneous symbolic and concrete execution
• Use concrete values to simplify highly complex
  constraint equations
• Partial order of events for processes
• Distributed programs are nondeterministic when
  scheduling processes
• Messages between processes are often delayed

7
Solution

• Algorithm to test distributed programs
• Use concolic testing to control the exeuction
  schedules
• Guide the execution path at the constraint
  locations using symbolic data
• Since algorithm uses concrete data, any bugs
  reported are real
• Algorithm is complete (in coverage) if all
  constraints can be handled by constraint solver

8
Distributed Program Model

• Asynchronous message-passing concurrent language
  (MPIL)
• Processes execute a sequence of statements
• Processes execute through asynchronous message
  passing
• All processes terminate or there are deadlocks
• New processes cannot be created during execution
• Variables must always be integers

9
Distributed Program Model

• Initial statement of every process begins with a
  RECEIVE statement
• Variables are always local to the process
• Processes communicate using SEND primitive
• Processes store incoming messages in queues
• Processes receive messages using RECEIVE
  primitive
• Processes wait if message queue is empty,
  non-deterministic when selecting message
• Process is active if hasnt be HALTed or ERROR
• Process is enabled if message queue is non-empty
  and next statement is RECIEVE

10
Macro-Step Execution

• A macro-step is the execution of a process from a
  RECEIVE statement to the next RECEIVE statement
  (not inclusive) consecutively without
  interleaving
• Macro-step execution non-deterministicly picks
  enabled processes and messages
• However, we can lexicographically (lt) order
  process and message pairs
• Using process and message index numbering
• Define a next function to choose next (process,
  message) pair

11
Happens-Before Relationship

• Define a relation (similar to lt) called
  happens-before for partial order events
• Can order events (process statement executions)
  in order based upon when they are executed
• Use an event associated vector clock to track
  happens-before relationship
• Can determine if two executions of a sequence of
  events are causally equivalent if they have the
  same happens-before relationship

12
Simple Algorithm

• Explores all macro-step execution paths
• Done by varying inputs based upon program
  constraints
• Done by creating new schedules at decision points
• Performs both symbolic and concrete execution
• All constraints at branch points are collected
• Records (process, message) pairs and symbolic
  constraints
• Backtracks using DFS to alter path at decision
  points (constraints, schedule choices)
• Constraints are negated and new inputs are
  calculated
• Picks next process schedule using next function
• Continues until no more enabled processes
• Deadlock occurs if not all processes finished

13
Example Distributed Program

• What do we need to do?
• Generate inputs and execute symbolically and
  concretely
• Find states where we can make decisions, at
  backtrack points
• Selecting p2 or p3, p3 1 RECIEVE, p3 2
  constraint, p3 4 RECIEVE
• For our decisions, we either generate a new input
  or a new schedule
• Explore execution path tree using a depth-first
  search.

14
Exampled Distributed Program

• Different executions with varying inputs and
  executions based upon backtrack locations (s1,
  s2, s3)
• Notice numerous duplicate executions, solved by
  taking into account happens-before relationship
  between events and processes

15
Efficient Algorithm

• At a scheduling decision point, use
  happens-before relationship to determine if only
  have to consider messages from a single process
  or all processes.
• Avoids exploring equivalent execution paths
• Use vector clocks to track happens-before
  relationship for each RECEIVE and SEND statement
• Flag RECEIVE event if happens after SEND event
  for process
• If flagged need to consider all messages from
  other processes

16
Implementation

• Implemented in Java tool, jCUTE
• Concolic testing engine for Java
• Generates test cases for all execution paths by
  analysis of source code
• Any bugs found is traceable since we know the
  execution path from the test

17
Results

• Any bug reported by the algorithms is an actual
  bug since algorithm provides concrete inputs and
  the schedule for which it exhibits the bug
  (similar to EXE)
• The algorithm is complete if
• The algorithm terminates
• Algorithm does not need to approximate any
  constraint during concolic execution

18
Results

• Used jCUTE to test on sample algorithms
• Leader election, distributed sorting,
  shortest-path
• Efficient algorithm explores significantly fewer
  execution paths with same branch coverage than
  simple algorithm
• Branch coverage less than 100 due to assert
  statements and dead branches

19
Conclusions

• Explores all execution paths of distributed
  programs
• Efficient algorithm eliminates equivalent
  execution paths using happens-before relationship
• Produces actual bugs since provides concrete
  input and schedule that result in the error
• When used with jCute provides test cases for all
  execution paths
• http//osl.cs.uiuc.edu/ksen/cute/

20
Thank You !