ConcJUnit: Unit Testing for Concurrent Programs

1
ConcJUnitUnit Testing for Concurrent Programs

• PPPJ 2009
• Mathias Ricken and Robert Cartwright
• Rice University
• August 28, 2009

2
Concurrency in Practice
Brian Goetz, Java Concurrency in Practice,
Addison-Wesley, 2006
3
Concurrency Practiced Badly
1
4
Unit Tests

• Occur early
• Automate testing
• Keep the shared repository clean
• Serve as documentation
• Prevent bugs from reoccurring
• Allow safe refactoring

5
Existing Unit Testing Frameworks

• JUnit, TestNG
• Dont detect test failures in child threads
• Dont ensure that child threads terminate
• Tests that should fail may succeed

6
ConcJUnit

• Replacement for JUnit
• Backward compatible, just replace junit.jar file
• Available at www.concutest.org
• Detects failures in all threads
• Warns if child threads outlive main thread
• Warns if child threads are not joined

7
Sample JUnit Tests

• public class Test extends TestCase
• public void testException()
• throw new RuntimeException("booh!")
• public void testAssertion()
• assertEquals(0, 1)

Both tests fail.
Both tests fail.

if (0!1) throw new AssertionFailedError()
8
JUnit Test with Child Thread
Main thread

• public class Test extends TestCase
• public void testException()
• new Thread()
• public void run()
• throw new RuntimeException("booh!")
• .start()

new Thread() public void run() throw
new RuntimeException("booh!") .start()
throw new RuntimeException("booh!")
Child thread
end of test
spawns
Main thread
success!
uncaught!
Child thread
9
JUnit Test with Child Thread

• public class Test extends TestCase
• public void testException()
• new Thread()
• public void run()
• throw new RuntimeException("booh!")
• .start()

new Thread() public void run() throw
new RuntimeException("booh!") .start()
throw new RuntimeException("booh!")
Uncaught exception, test should fail but does not!

• By default, no uncaught exception handler
  installed for child threads

10
Changes to JUnit (1 of 3)

• Thread group with exception handler
• JUnit test runs in a separate thread, not main
  thread
• Child threads are created in same thread group
• When test ends, check if handler was invoked
• Reasoning
• Uncaught exceptions in all threads must cause
  failure

11
JUnit Test with Child Thread

• public class Test extends TestCase
• public void testException()
• new Thread()
• public void run()
• throw new RuntimeException("booh!")
• .start()

new Thread() public void run() throw
new RuntimeException("booh!") .start()
throw new RuntimeException("booh!")
spawns and joins
resumes
Main thread
failure!
check groups handler
end of test
Test thread
uncaught!
invokes groups handler
Child thread
12
Child Thread Outlives Parent

• public class Test extends TestCase
• public void testException()
• new Thread()
• public void run()
• throw new RuntimeException("booh!")
• .start()

new Thread() public void run() throw
new RuntimeException("booh!") .start()
throw new RuntimeException("booh!")
check groups handler
Main thread
success!
Test thread
Too late!
end of test
uncaught!
invokes groups handler
Child thread
13
Changes to JUnit (2 of 3)

• Check for living child threads after test ends
• Reasoning
• Uncaught exceptions in all threads must cause
  failure
• If the test is declared a success before all
  child threads have ended, failures may go
  unnoticed
• Therefore, all child threads must terminate
  before test ends

14
Check for Living Child Threads

• public class Test extends TestCase
• public void testException()
• new Thread()
• public void run()
• throw new RuntimeException("booh!")
• .start()

new Thread() public void run() throw
new RuntimeException("booh!") .start()
throw new RuntimeException("booh!")
check for living child threads
check groups handler
Main thread
failure!
Test thread
end of test
uncaught!
invokes groups handler
Child thread
15
Correctly Written Test

• public class Test extends TestCase
• public void testException()
• Thread t new Thread()
• public void run() / child thread /
• t.start()
• t.join()

Thread t new Thread() public void run()
/ child thread / t.start() t.join() //
wait until child thread has ended
/ child thread /
check for living child threads
check groups handler
Main thread
success!
Test thread
end of test
Child thread
4
16
Changes to JUnit (3 of 3)

• Check if any child threads were not joined
• Reasoning
• All child threads must terminate before test ends
• Without join() operation, a test may get lucky
• Require all child threads to be joined

17
Fork/Join Model

• Parent thread joins with each of its child
  threads
• May be too limited for a general-purpose
  programming language

Main thread
Child thread 1
Child thread 2
18
Example of Other Join Models

• Chain of child threads guaranteed to outlive
  parent
• Main thread joins with last thread of chain

Main thread
Child thread 1
Child thread 2
Child thread 3
19
Generalize to Join Graph

• Threads as nodes edges to joined thread
• Test is well-formed as long as all threads are
  reachable from main thread

Main thread
MT
Child thread 1
CT1
Child thread 2
CT2
Child thread 3
CT3
20
Unreachable Nodes

• An unreachable node has not been joined
• Child thread may outlive the test

Main thread
MT
Child thread 1
CT1
Child thread 2
CT2
21
Constructing the Graph start()

• // in mainThreadchildThread.start()
• Add node for childThread

main Thread
MT
childThread
CT
22
Constructing the Graph join()

• // in mainThreadchildThread.join()
• When leaving join(), add edge from mainThread to
  childThread

main Thread
MT
child Thread
CT
3
23
Modifying the Java Runtime

• Changing Thread.start()and join()
• Need to modify Java Runtime Library
• Utility to process users rt.jar file
• Put new jar file on boot classpath-Xbootclasspat
  h/pnewrt.jar
• Still works without modified Thread class
• Just does not emit lucky warnings

24
Evaluation

• JFreeChart
• All tests passed tests are not concurrent
• DrJava 900 unit tests
• Passed 880
• No join 1
• Lucky 18
• Timeout 1
• Runtime overhead 1 percent

25
Limitations

• Only checks chosen schedule
• A different schedule may still fail
• Example
• Thread t new Thread()if (nondeterministic())
  t.join()

2
26
Future Work

• Insert sleeps or yields in critical code
  locations
• Example If a notify() is delayed, a wait()
  may time out.
• Can detect a number of sample problems
• Run tests several times on build server
• Record schedule, replay if test fails
• Makes failures reproducible if found

5
27
ConcJUnit Demo
28
Thank you!
www.concutest.org
29
Notes
30
Notes

• Probably not caused by concurrency problems just
  a metaphor. ?
• Also cannot detect uncaught exceptions in a
  programs uncaught exception handler (JLS
  limitation) ?
• Only add edge if joined thread is really dead do
  not add if join ended spuriously. ?

31
Spurious Wakeup

• public class Test extends TestCase
• public void testException()
• Thread t new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• )
• t.start()
• while(t.isAlive())
• try t.join()
• catch(InterruptedException ie)

Thread t new Thread(new Runnable() public
void run() throw new RuntimeException("booh!
") ) t.start() while(t.isAlive()) try
t.join() catch(InterruptedException ie)

throw new RuntimeException("booh!")
Loop since join() may end spuriously
?
32
Notes

• Have not studied probabilities or durations for
  sleeps/yieldsOne inserted delay may negatively
  impact a second inserted delayExample If both
  notify() and wait() are delayed. ?

33
Extra Slides
34
JUnit Test with Child Thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
invokes
checks
TestGroups Uncaught Exception Handler
35
Join with All Offspring Threads

• Main thread joins with all offspring threads,
  regardless of what thread spawned them

Main thread
Child thread 1
Child thread 2
36
Join Graph Examples
Main thread
MT
Child thread 1
CT1
Child thread 2
CT2
Main thread
MT
Child thread 1
CT1
Child thread 2
CT2
37
Thread Creation Context

• In Thread.start() also record stack trace of
  Thread.currentThread(
• Easy to find source code of a child thread that
  is not joined
• Also available for uncaught exception stack traces

38
Creation Context Example
AssertionError at Helper.m(Helper.java2) at
Helper.run(Helper.java3) Started at at
Main.foo(Main.java4) at Main.bar(Main.java15) at
Main.main(Main.java25)
class Helper extends Thread void m()
Assert.fail() public void run() m()

• class Main
• void foo()
• // which one?
• new Helper().start()
• new Helper().start()
• // ...

39
Many Thanks To

• My advisor
• Corky Cartwright
• My committee members
• Walid Taha
• David Scott
• Bill Scherer
• NSF and Texas ATP
• For providing partial funding

40
Concurrency Problems NotFound During Testing
1