Test Driven Development

1
Test Driven Development Testing the Graphical
User Interface

• Prepared by
• Yibo Sun
• Soha Makady

2
Agenda

• What is GUI testing?
• Generation of test cases
• Running test cases
• Design approaches for TDD of GUI
• GUI TDD problems
• Programmatic GUI Testing
• Conclusion

3
Definition

• The process of testing a software product with
  graphical user interface
• Like testing any other kind of codes, graphical
  user interface testing requires test cases
• All GUI functionalities of the SUT should be
  covered by test cases

4
Importance

• GUI accounts for as much as 60 of the total
  application code
• Use in safety critical systems

5
Benefits

• Safety
• Robustness
• Usability

6
Difficulty

• Scope huge number of GUI operations, states
  which need to be tested
• Sequence GUI operations are dependent and rely
  on other operations, when the sequence is very
  long, the test case will be very long and error
  prone.
• Regression Test GUI may change significantly
  across versions, test cases need to be changed
  repeatedly

7
Coverage Criteria

• Conventional software testing uses coverage
  criteria which define a set of rules telling the
  type and amount of code to test
• It is hard to map GUI states and events to
  underlying code
• GUI states
• It is hard to determine the most important subset
  of all possible test cases

8
Regression A.F. Memon and M.L. Soffa,
Regression testing of GUIs, Software
Engineering Notes, Sep. 2003

• Test cases become obsolete
• Recognize GUI components by names or IDs instead
  of their placements
• 74 of test cases are broken per version (study
  on Adobe Reader 4.0 and 5.0)

9
Generation of Test Cases

• Manual
• Automated
• Other Monkey Testing

10
Automated Test Case Generation

• Approaches for automated test case generation
• State machine modelling
• Automated Planning System
• Genetic Algorithm

11
Automated Test Case Generation - State Machine
Modelling
Event/Transition 1
State 1
State 2
Event/Transition 2
Event/Transition 3
State 3
State 4
12
State Machine Modelling

• Modelling GUI as a state machine with states and
  events (transitions)
• FSM (Finite State Machine)
• Scaling problem
• VFSM (Variable Finite State Machine)
• Assertion for GUI states to be inserted manually,
  at points determined by test designers

13
Automated Planning System

• Based on an artificial intelligence algorithm
• Defines
• an initial state
• a goal state
• a set of operations
• A test plan is determined as a path from an
  initial state to a goal state
• Testers focus on what function to test instead of
  how to test a function which is determined by the
  test plan path
• Limitation Manually decide the tasks (initial
  and goal states) to be tested and poorly chosen
  tasks can produce a test suite without adequate
  GUI testing coverage

14
Genetic Algorithm

• Generate tests simulating novice usage, a more
  random path achieves more system states
• A gene is a path through GUI states transitions
  generated for some task
• Choose an initial gene to mimic a novice path.
• More genes are then generated by modifying and
  extending the initial gene
• Genes that complete tasks best are kept, scored
  by some criteria as best novice paths.
• Surviving genes are replicated and the natural
  selection process is repeated with more genes.
• Eventually, multiple test cases to mimic novice
  paths (genes) are produced
• Limitation The initial gene has to be chosen
  manually, and thus this approach largely relies
  on the human experts ability to choose the
  initial gene.

15
What is a Good GUI Suite? Xie and Memon,
ISSRE, 2006

• Three experiments were conducted to answer these
  questions
• How many test cases should be generated?
• What should be the length of each test case?
• What events should be put into a test case?
• Tested applications TerpWord, TerpSpreadSheet,
  TerpPaint, and TerpCalc
• The primary measured variable is the
  fault-detection effectiveness of a test suite

16
Used Terminology in Experiments

• Test case A sequence of events
• Fault A mismatch between the expected and actual
  GUI states (asserted for every GUI property in a
  certain GUI state)
• Fault-detection effectiveness the number of
  unique faults detected by the test cases in the
  suite

17
Effect of Test Suite Size
For all suites Fixed test case length Fixed
event composition
Largest observation
Upper quartile
Median
Lower quartile
Smallest observation
Fault Detection Effectiveness vs. Test Suite Size
for TerpCalc
18
Effect of Test Case Length
For all suites Fixed suite size Fixed event
composition
Fault Detection Effectiveness vs. Test case
length Size for TerpCalc
19
Effect of Event Composition

• Fixed test suite size
• Fixed test case length
• Results revealed that absence of an event (that
  interacted with a functional unit ) in a test
  suite directly effects the detection of a fault
• Example When running a test suite that does not
  contain a click event on a form full of
  buttons, not all faults are detected

20
Running GUI Test Cases

• Generated test cases (manually/automatically) can
  be run in two ways
• Playing scripts using Record playback
  technique
• Programmatically send commands/events using API
  functions

21
Record and Playback

• Capture events and put states assertions in
  scripts
• Playback events in scripts
• Compare expected state in scripts to actual UI
  state to determine the correctness of execution
• Scripts can be edited when UI is changed

22
Record and Playback Tools

• Java AWT and Swing
• Marathon
• Jacareto
• Abbot(Costello)
• Web
• Selenium

23
Record and playback tools - Comparison
24
Selenium

• Runs directly in a browser
• Supports web page testing

25
Demo
26
Drawbacks of Record playback

• Requires the GUI to be built first
• Assumes the GUI under test can already be
  functional
• The generated test scripts cannot keep up with
  design changes

27
GUI Architectures for TDD

• Following, are two proposed GUI architectures to
  enable GUI unit testing
• The Humble Dialog Box
• Presenter First

28
The Humble Dialog BoxMichael Feathers, Object
Mentor, Inc. 2002

• Creates smart objects that contain the functional
  behavior of a particular GUI object
• Adds a thin GUI view object (a humble dialog) to
  display the smart objects information (and has
  no application logic)
• Tests the smart objects mainly
• As a result, the application behavior will be
  covered by unit tests, and the display code (GUI)
  will not require complex GUI tests

29
The Humble Dialog -ExamplePaul Hamill, Unit Test
Frameworks, OReilly , 2004

• Consider a Library GUI application that allows
  the user to add new books.
• GUI design sketch for the AddBook dialog Window
• The smart object AddBook and its humble dialog
  (view class) AddBookView

30
The Humble Dialog Box Example (Contd)

• Unit test for the smart object AddBook
• public class AddBookTest extends TestCase
  public void testAddBook( )
• Library library new Library( )
• AddBook addBook new AddBook(library)
  addBook.add(" Java ", "Carl Sagan")
• assertNotNull( library.getBooksByTitle(" Java
  ") )

31
The Humble Dialog Box Example (Contd)

• Unit test for the humble dialog
  AddBookViewTest.java
• public class AddBookViewTest extends TestCase
• public void setUp( )
• public void testControlValues( )
• AddBookView view new AddBookView( )
  assertEquals( "Add Book", view.getTitle( ) )
  assertEquals( "", view.titleField.getText( ) )
  assertEquals( "", view.authorField.getText( ) )
  assertEquals( "Add", view.addButton.getText( )
  ) assertEquals( "Cancel", view.cancelButton.getT
  ext( ) )

Testing application GUI
32
The Humble Dialog Box Example (Contd)

• Unit test for the humble dialog
  AddBookViewTest.java
• public class AddBookViewTest extends TestCase
• public void testAddButton( )
• view.titleField.setText(Java")
  view.authorField.setText("Carl Sagan")
  view.addButton.doClick( )
• assertEquals(1, library.getBooksByTitle(Java").
  size( ))

Testing application functionality
33
Drawbacks of the Humble Dialog Box Approach

• Tested GUI components have to be visible as
  non-private instance variables, or through getter
  methods, which conflicts with the normal design
  convention, and increases GUI code.
• GUI components do not operate over the actual
  event mechanism, thus not all errors will be
  revealed.
• Example Invisible and disabled components can be
  addressed in the unit tests.

34
Presenter First (PF)Micah Alles et al.,
Presenter First, AGILE06

• A design and development technique to produce
  fully tested GUI applications following TDD
• A variant of the Model View Presenter (MVP)
  design pattern

35
Model View Presenter Pattern

• Model The data and logic for business aspects
• View what the user sees and interacts with
• Presenter Behavior that corresponds to customer
  stories (application writing)

Any interaction between View and Model takes
place through Presenter
36
MVP Variant for PF
Dashed lines are events, solid lines are object
messages

• Model and View are isolated from each other
• Model and View are members of Presenter, and do
  not know of its presence
• Presenter works with Model and View interfaces
• Testing the View can be automated using tools, or
  done manually

37
MVP Variant for PF (Contd)
38
PF Example Presenter Class

• public class TBPresenter
• private ITBModel m_model
• private ITBView m_view
• public TBPresenter(ITBModel model, ITBView view)
  
• m_model model
• m_view view
• m_view.SubscribeItemDrag(new
  UpdateDelegate(ItemDrag))
• m_model.SubscribeToolboxItemsChanged(new
• UpdateDelegate(ToolboxItemsChanged))

39
The Burke Porter Dyno Host Project
http//atomicobject.com/pages/BurkePorterDynoH
ost

• Done in Presenter First Style
• Status
• 6 developers for 9 months
• One week iterations
• 99 estimation accuracy over project lifetime
• Full test driven development (5200 automated
  unit tests)
• Project A Visually configured automotive test
  and measurement application
• But
• The level of experience of the developers is not
  mentioned as a factor in the estimates

40
GUI TDD Unit Testing Problems

• Difficulty in verifying some behaviors
• Example Check that the dialog box pops up
  modally showing the alert icon and the warning
  message in red.)
• Using ONLY JUnit has its limitations
• Tests may start before the application starts as
  they run in a separate thread
• When starting the application from within the
  unit test, it may take time loading, thus the
  test may fail due to the time out waiting for the
  GUI
• There is no easy way to find and interact with
  GUI components except through Java Robot class

41
java.awt.Robot Class

• Emulates end-user interactions using the keyboard
  and/or mouse.
• Limitations
• Takes control of the mouse and keyboard, so the
  user cannot do anything else on the computer
  while the test is running
• Tests are fragile, where any change in the layout
  will break them
• The user would have to add getter methods, or
  loop through the component hierarchy to reference
  a certain component.

42
Programmatic GUI Testing Tools (Comparison)
43
Abbot Java GUI Test Framework

• Supports record and playback (Costello) and
  programmatic handling
• Supports both AWT and Swing components
• Based on Java AWT Robot
• Main components
• A single finder class
• A tester class for one/several components

44
Demo
45
Conclusion

• Record and playback tools cannot be used when
  developing a GUI following TDD
• Designing an application following the PF style,
  makes it easier to follow a TDD approach.
• Testing the application GUI (view) can be done
  through a programmatic GUI testing tool that can
  find components regardless of their
  layout/position
• There is still no clear criteria for determining
  the GUI testing coverage (e.g. no tool like
  Coverlipse for GUI testing coverage)
• Unit testing GUI scenarios is complicated due to
  the need to simulate many actions to reach the
  initial state for a certain test case

46
References

• A.F. Memon, GUI testing pitfalls and process,
  Computer, v 35, n 8, Aug., pp. 87-88, 2002
• A.F. Memon and M.L. Soffa, Regression testing of
  GUIs, Software Engineering Notes, v 28, n 5,
  Sept., pp. 118-27, 2003
• R.K. Shehady and D.P. Siewiorek, A Method to
  Automate User Interface Testing Using Variable
  Finite State Machines, Proc. 27th Ann. Int'l
  Symp. Fault-Tolerant Computing (FTCS '97), pp.
  80-88, June 1997
• A.F. memon, M.E. Pollock, and M.L. Soffa,
  Hierarchical GUI test case generation using
  automated planning, IEEE Transaction on Software
  Engineering, Vol. 27, Issue 2, Feb., pp. 144-55,
  2001.

47
References (Contd)

• D.J. Kasik and H.G. George, Toward Automatic
  Generation of Novice User Test Scripts, Proc.
  Conf. Human Factors in Computing Systems Common
  Ground, M.J. Tauber, V. Bellotti, R. Jeffries,
  J.D. Mackinlay, and J. Nielsen, eds., pp.
  244-251, Apr. 1996
• Jacareto, http//jacareto.sourceforge.net/
• Abbot, http//abbot.sourceforge.net/doc/overview.s
  htm
• Selenium, http//www.openqa.org/selenium/.
• Marathon, http//www.marathon.com
• JFCUnit, http//jfcunit.sourceforge.net/
• MVC and MVP patterns http//www.martinfowler.com

48
Thank you