Testing Multithreaded Java Programs

1
Testing Multi-threaded Java Programs
Shmuel Ur
2
Outline

• It started with a bug
• Finding concurrent bugs is hard
• More bugs
• ConTest
• Philosophy
• Demo

3
Our Real Life Motivation

• Project File system (IFS for the AS/400)
• Stage Component test
• Observed behavior "Use-count" values corrupted
  during stress test
• Time wasted Several calendar months, more than
  0.5 PY
• Main difficulty Failure was not repeatable
• Fault description Incorrect use of
  compare-and-swap result
• Bug should have been found during the unit test!

4
Why is Concurrent Testing Hard?

• Concurrency introduces non-determinism
• Multiple executions of the same test may have
  different interleaving (and different results!)
• an interleaving is the relative execution order
  of the program threads
• Re-executing a test on a single stand-alone
  processor is not useful
• Debugging affects the timing
• No useful coverage measures for the interleaving
  space
• Result Most bugs are found in system tests,
  stress tests, or by the customer

5
Example of an Interleaving-sensitive Test Result

• Threads A and B execute (add(i))
• The result depends upon the interleaving
• This would not be revealed in a typical test

Interleaving
Program
A1 A2 A3 B1 B2 B3 2
A1 B1 B2 A2 B3 A3 1
add(int x) 1 int tmp x 2
tmp 3 x tmp
Result
6
Version control problems
Scenario 1
Bob and Julie are working on a shared document
7
Bob makes a copy of the document
8
Bob makes changes to the copy
9
Bob saves the copy in the shared space. He saves
over the original document.
10
Julie makes a copy of the updated document
11
Julie makes changes to the document
12
Julie saves her copy in the shared space.
13
Both sets of changes are saved.
14
Scenario 2
Bob makes a copy of the document
15
Bob makes changes to his copy
16
Julie makes a copy of the document
17
Julie makes changes to her copy
18
Bob saves his copy in the shared space
19
Julie saves her copy in the shared space.
20
Julie has saved her changes over Bobs changes.
21
How Much is 110100100010000?
static final int NUM 5 static int Global
0 public static void main(String argv)
Global 0 threads0 new Adder(1)
threads1 new Adder(10) threads2 new
Adder(100) threads3 new Adder(1000)
threads4 new Adder(10000) // Start
Threads for(int i0iltNUMi)
threadsi.start() try
Thread.sleep(1000) catch(Exception exc)
// Print Results System.out.println(Globa
l)
22
class Adder
class Adder extends Thread int Local
Adder(int i) Local i public
void add() example.Global Local
public void run() add()
23
Coverage Results Without ConTest
24
Coverage Results with ConTest
25
method Add
Java Source code public void add()
example.Global Local Byte Code Method
void add() 0 getstatic 3 ltField int Globalgt 3
aload_0 4 getfield 2 ltField int Localgt 7 iadd
8 putstatic 3 ltField int Globalgt 11 return
26
A Bug We Published...
27
Creating a Critical Section is Tricky
class ObjectOne implements Runnable public
void run() if(RaceField.race false)
RaceField.race true
System.out.println("Mine") class
RaceField public static boolean race
false Expected and observed Behavior One
"Mine" is printed by the thread that won the
race Possible bug More than one "Mine" is
printed
28
Atomicity is Never Ensured

• static void transfer(Transfer t)
• balancest.fundFrom - t.amount
• balancest.fundTo t.amount
• Expected Behavior
• Money should pass from one account to another
• Observed Behavior
• Sometimes the amount taken is not equal to the
  amount received
• Possible bug
• Thread switch in the middle of money transfers

29
Bug Found by ConTest in Websphere Site Analyzer

• Crawler, a 1000 line Java component, is part of
  the Websphere Site Analyzer used to perform
  content analysis of web sites
• Bug description
• if (connection ! null) connection.setStopFlag()
• Connection is checked to be !null
• CPU is lost
• Connection is set to null before CPU is regained
  
• If this happens before connection.setStopFlag()
  is executed, an exception is taken
• This bug was found while we were still testing
  ConTest
• This bug should (also) have been found in unit
  testing...

30
A Servlet Server Bug (HA. Tomcat)

• A high availability servlet server has a hash
  table that is kept synchronized using a cluster
• The high availability servlet has a primary and a
  backup node
• The primary node caches the hash table
• Objects are trees deep copy is obtained using an
  auxiliary table

hash table
auxiliary table
cache
primary
31
A Servlet Server Bug

• Bug description
• thread1
• A node becomes primary
• Start deep copying of hash table to CACHE table
  using the auxiliary table
• thread2
• A cache miss on object A occurs
• Start deep copying of object A from hash table to
  CACHE table using auxiliary table
• thread1
• Complete deep copying of hash table to the CACHE
  table. Auxiliary table is discarded
• thread2
• An attempt is made to access the auxiliary table

32
Why are These Bugs Not Found?

• Frame of mind when the program is written
• Requires thread switching at precise locations
• Typical testing environment
• Thread switch occurs at repeating locations
• Execution is almost deterministic
• No load/stress
• Not enough of the right kind of tests
• Not enough tests

33
How Does ConTest Find Bugs?

• We instrument every concurrent event
• Concurrent events are the events whose order
  determines the result of the program
• At every concurrent event, a random based
  decision is made whether to cause a context
  switch
• For example, using a sleep statement
• Philosophy
• Modify the program in such a way that it will be
  more likely to exhibit bugs (without introducing
  new bugs)
• Minimize impact on the testing process (under the
  hood technology)
• Re-use existing tests
• Utilize idle computer time

34
ConTest Overview

• ConTest is composed of the following components
• An instrumentation engine that
• Creates hooks for the irritator and for coverage
  printing
• Generates coverage models
• The instrumentation is done at the byte code
  level
• An irritator that randomly, or using heuristics,
  generates new interleaving on-the-fly
• Seed replay component
• Coverage component
• Debugging aids

35
ConTest Architecture - Static View
Application under test
Optional
Coverage Model Replay Debugging Support
Instrumentation
Application irritator instrumentation
36
ConTest Architecture - Dynamic View
Test
Optional
Coverage information
Execute instrumented program
Replay information
Deadlock Orange box
Results
37
User Scenario Before ConTest
Run Test
Checkresults
Problem
Correct
Fix bug
Finish
38
User Scenario After ConTest
Run test with interleaving decided by
heuristic Record interleaving Update coverage
Check results
Problem
Correct
Fix bug Rerun test with replay information Orange
box Deadlock
Not Reached
Check coverage target
Reached
Finish
39
Design Consideration - Replay

• Full Replay
• Design measure every interleaving and replay
• Source of randomness
• Random, Date
• Inputs, files, OS functions
• Hash functions, other non deterministic functions
• Multi-threading
• Exceptions
• Use algorithm by Jong-Deok Choi, Harini
  Srinivasan
• Partial replay use seed for replay

40
Design Consideration - Instrumentation

• Source code
• Portable (in other languages)
• Complex
• Problems with atomicity e.g. float
• Tool - JIE
• Byte Code
• Simple
• Portable
• Tool - CFParse

41
Design Consideration Instrumentation II

• Where not to instrument
• Location where only one thread exist
• Location where only one thread is active
• Areas protected by synchronization?
• Protected variables?
• Bug patterns
• Coverage and instrumentation

42
Why Bugs are Found with ConTest

• Thread switches in many places
• Random interleaving the interleaving changes
  between runs of the same test
• Relevant coverage tells you what functionality
  needs more testing
• Result executing tests many times is more likely
  to find bugs

43
With ConTest, each test will go a long way
44
Heuristics

• Noise
• Type yields, sleeps, synchYields,random (default
  random)
• Type is the mechanism that forces thread switches
• Strength integer (default -1)
• Strength effect the desire not to return to the
  thread
• StartLate
• valueclass,method,off (default off)
• Name string (default empty)
• Field problem skip bugs in initialization
• Solution start ConTest late

45
Heuristics II

• HaltOneThreadfalse,true (default false)
• Stop a thread until no other is willing to run
• SharedNoiseParameters
• sharedVarNoisefalse,true (default false)
• sharedNoiseall,one (defalt all)
• collectSharedfalse,true (default true)
• nonVarNoisefalse,true (default false)
• Only shared variables are interesting
• Useful for finding bugs and for evaluating tests

46
Coverage

• Everyone wants coverage
• Coverage is different for testers than developers
• Testers usually look at higher level coverage
  models
• ConTest is tester oriented
• Implementation is by measuring which
  instrumentation where exercised
• Performance impact is larger if we add
  instrumentation points

47
Coverage II

• methodCoverage every method
• branchCoverage every branch/block
• concurrentEventCoverage coverage of existing
  instrumentation point
• uncaughtExceptionCoverage for methods
• sharedVarCoverage - variables touched by more
  then one thread
• concurrentEventPairsCoverage Consecutive
  instrumentation points

48
Communication Parameters

• enable_keyboard_callback talk to the
  application from the keyboard
• enable_port_callback talk to the application
  from a port
• Used to enable debugging!

49
Debug Parameters

• Deadlock
• Where is each thread
• Cycle of waiting on locks
• orange_box remember last_values_size of
  values and locations for each variable. Created
  for null pointer exception
• replay record and reuse replay information

50
Find information on ConTest at
http//www.haifa.il.ibm.com/projects/verification/
contest/index.html