Title: Test Driven Development
1Test Driven Development Testing the Graphical
User Interface
- Prepared by
- Yibo Sun
- Soha Makady
2Agenda
- What is GUI testing?
- Generation of test cases
- Running test cases
- Design approaches for TDD of GUI
- GUI TDD problems
- Programmatic GUI Testing
- Conclusion
3Definition
- The process of testing a software product with
graphical user interface - Like testing any other kind of codes, graphical
user interface testing requires test cases - All GUI functionalities of the SUT should be
covered by test cases
4Importance
- GUI accounts for as much as 60 of the total
application code - Use in safety critical systems
5Benefits
- Safety
- Robustness
- Usability
6Difficulty
- Scope huge number of GUI operations, states
which need to be tested - Sequence GUI operations are dependent and rely
on other operations, when the sequence is very
long, the test case will be very long and error
prone. - Regression Test GUI may change significantly
across versions, test cases need to be changed
repeatedly
7Coverage Criteria
- Conventional software testing uses coverage
criteria which define a set of rules telling the
type and amount of code to test - It is hard to map GUI states and events to
underlying code - GUI states
- It is hard to determine the most important subset
of all possible test cases
8Regression A.F. Memon and M.L. Soffa,
Regression testing of GUIs, Software
Engineering Notes, Sep. 2003
- Test cases become obsolete
- Recognize GUI components by names or IDs instead
of their placements - 74 of test cases are broken per version (study
on Adobe Reader 4.0 and 5.0)
9Generation of Test Cases
- Manual
- Automated
- Other Monkey Testing
10Automated Test Case Generation
- Approaches for automated test case generation
- State machine modelling
- Automated Planning System
- Genetic Algorithm
11Automated Test Case Generation - State Machine
Modelling
Event/Transition 1
State 1
State 2
Event/Transition 2
Event/Transition 3
State 3
State 4
12State Machine Modelling
- Modelling GUI as a state machine with states and
events (transitions) - FSM (Finite State Machine)
- Scaling problem
- VFSM (Variable Finite State Machine)
- Assertion for GUI states to be inserted manually,
at points determined by test designers
13Automated Planning System
- Based on an artificial intelligence algorithm
- Defines
- an initial state
- a goal state
- a set of operations
- A test plan is determined as a path from an
initial state to a goal state - Testers focus on what function to test instead of
how to test a function which is determined by the
test plan path - Limitation Manually decide the tasks (initial
and goal states) to be tested and poorly chosen
tasks can produce a test suite without adequate
GUI testing coverage
14Genetic Algorithm
- Generate tests simulating novice usage, a more
random path achieves more system states - A gene is a path through GUI states transitions
generated for some task - Choose an initial gene to mimic a novice path.
- More genes are then generated by modifying and
extending the initial gene - Genes that complete tasks best are kept, scored
by some criteria as best novice paths. - Surviving genes are replicated and the natural
selection process is repeated with more genes. - Eventually, multiple test cases to mimic novice
paths (genes) are produced - Limitation The initial gene has to be chosen
manually, and thus this approach largely relies
on the human experts ability to choose the
initial gene.
15What is a Good GUI Suite? Xie and Memon,
ISSRE, 2006
- Three experiments were conducted to answer these
questions - How many test cases should be generated?
- What should be the length of each test case?
- What events should be put into a test case?
- Tested applications TerpWord, TerpSpreadSheet,
TerpPaint, and TerpCalc - The primary measured variable is the
fault-detection effectiveness of a test suite
16Used Terminology in Experiments
- Test case A sequence of events
- Fault A mismatch between the expected and actual
GUI states (asserted for every GUI property in a
certain GUI state) - Fault-detection effectiveness the number of
unique faults detected by the test cases in the
suite
17Effect of Test Suite Size
For all suites Fixed test case length Fixed
event composition
Largest observation
Upper quartile
Median
Lower quartile
Smallest observation
Fault Detection Effectiveness vs. Test Suite Size
for TerpCalc
18Effect of Test Case Length
For all suites Fixed suite size Fixed event
composition
Fault Detection Effectiveness vs. Test case
length Size for TerpCalc
19Effect of Event Composition
- Fixed test suite size
- Fixed test case length
- Results revealed that absence of an event (that
interacted with a functional unit ) in a test
suite directly effects the detection of a fault - Example When running a test suite that does not
contain a click event on a form full of
buttons, not all faults are detected
20Running GUI Test Cases
- Generated test cases (manually/automatically) can
be run in two ways - Playing scripts using Record playback
technique - Programmatically send commands/events using API
functions
21Record and Playback
- Capture events and put states assertions in
scripts - Playback events in scripts
- Compare expected state in scripts to actual UI
state to determine the correctness of execution - Scripts can be edited when UI is changed
22Record and Playback Tools
- Java AWT and Swing
- Marathon
- Jacareto
- Abbot(Costello)
- Web
- Selenium
23Record and playback tools - Comparison
24Selenium
- Runs directly in a browser
- Supports web page testing
25Demo
26Drawbacks of Record playback
- Requires the GUI to be built first
- Assumes the GUI under test can already be
functional - The generated test scripts cannot keep up with
design changes
27GUI Architectures for TDD
- Following, are two proposed GUI architectures to
enable GUI unit testing - The Humble Dialog Box
- Presenter First
28The Humble Dialog BoxMichael Feathers, Object
Mentor, Inc. 2002
- Creates smart objects that contain the functional
behavior of a particular GUI object - Adds a thin GUI view object (a humble dialog) to
display the smart objects information (and has
no application logic) - Tests the smart objects mainly
- As a result, the application behavior will be
covered by unit tests, and the display code (GUI)
will not require complex GUI tests
29The Humble Dialog -ExamplePaul Hamill, Unit Test
Frameworks, OReilly , 2004
- Consider a Library GUI application that allows
the user to add new books. - GUI design sketch for the AddBook dialog Window
- The smart object AddBook and its humble dialog
(view class) AddBookView
30The Humble Dialog Box Example (Contd)
- Unit test for the smart object AddBook
- public class AddBookTest extends TestCase
public void testAddBook( ) - Library library new Library( )
- AddBook addBook new AddBook(library)
addBook.add(" Java ", "Carl Sagan") - assertNotNull( library.getBooksByTitle(" Java
") ) -
31The Humble Dialog Box Example (Contd)
- Unit test for the humble dialog
AddBookViewTest.java - public class AddBookViewTest extends TestCase
- public void setUp( )
- public void testControlValues( )
- AddBookView view new AddBookView( )
assertEquals( "Add Book", view.getTitle( ) )
assertEquals( "", view.titleField.getText( ) )
assertEquals( "", view.authorField.getText( ) )
assertEquals( "Add", view.addButton.getText( )
) assertEquals( "Cancel", view.cancelButton.getT
ext( ) ) -
Testing application GUI
32The Humble Dialog Box Example (Contd)
- Unit test for the humble dialog
AddBookViewTest.java - public class AddBookViewTest extends TestCase
-
- public void testAddButton( )
- view.titleField.setText(Java")
view.authorField.setText("Carl Sagan")
view.addButton.doClick( ) - assertEquals(1, library.getBooksByTitle(Java").
size( )) -
Testing application functionality
33Drawbacks of the Humble Dialog Box Approach
- Tested GUI components have to be visible as
non-private instance variables, or through getter
methods, which conflicts with the normal design
convention, and increases GUI code. - GUI components do not operate over the actual
event mechanism, thus not all errors will be
revealed. - Example Invisible and disabled components can be
addressed in the unit tests.
34Presenter First (PF)Micah Alles et al.,
Presenter First, AGILE06
- A design and development technique to produce
fully tested GUI applications following TDD - A variant of the Model View Presenter (MVP)
design pattern
35Model View Presenter Pattern
- Model The data and logic for business aspects
- View what the user sees and interacts with
- Presenter Behavior that corresponds to customer
stories (application writing)
Any interaction between View and Model takes
place through Presenter
36MVP Variant for PF
Dashed lines are events, solid lines are object
messages
- Model and View are isolated from each other
- Model and View are members of Presenter, and do
not know of its presence - Presenter works with Model and View interfaces
- Testing the View can be automated using tools, or
done manually
37MVP Variant for PF (Contd)
38PF Example Presenter Class
- public class TBPresenter
- private ITBModel m_model
- private ITBView m_view
- public TBPresenter(ITBModel model, ITBView view)
- m_model model
- m_view view
- m_view.SubscribeItemDrag(new
UpdateDelegate(ItemDrag)) - m_model.SubscribeToolboxItemsChanged(new
- UpdateDelegate(ToolboxItemsChanged))
-
39The Burke Porter Dyno Host Project
http//atomicobject.com/pages/BurkePorterDynoH
ost
- Done in Presenter First Style
- Status
- 6 developers for 9 months
- One week iterations
- 99 estimation accuracy over project lifetime
- Full test driven development (5200 automated
unit tests) - Project A Visually configured automotive test
and measurement application - But
- The level of experience of the developers is not
mentioned as a factor in the estimates
40GUI TDD Unit Testing Problems
- Difficulty in verifying some behaviors
- Example Check that the dialog box pops up
modally showing the alert icon and the warning
message in red.) - Using ONLY JUnit has its limitations
- Tests may start before the application starts as
they run in a separate thread - When starting the application from within the
unit test, it may take time loading, thus the
test may fail due to the time out waiting for the
GUI - There is no easy way to find and interact with
GUI components except through Java Robot class
41java.awt.Robot Class
- Emulates end-user interactions using the keyboard
and/or mouse. - Limitations
- Takes control of the mouse and keyboard, so the
user cannot do anything else on the computer
while the test is running - Tests are fragile, where any change in the layout
will break them - The user would have to add getter methods, or
loop through the component hierarchy to reference
a certain component.
42Programmatic GUI Testing Tools (Comparison)
43Abbot Java GUI Test Framework
- Supports record and playback (Costello) and
programmatic handling - Supports both AWT and Swing components
- Based on Java AWT Robot
- Main components
- A single finder class
- A tester class for one/several components
44Demo
45Conclusion
- Record and playback tools cannot be used when
developing a GUI following TDD - Designing an application following the PF style,
makes it easier to follow a TDD approach. - Testing the application GUI (view) can be done
through a programmatic GUI testing tool that can
find components regardless of their
layout/position - There is still no clear criteria for determining
the GUI testing coverage (e.g. no tool like
Coverlipse for GUI testing coverage) - Unit testing GUI scenarios is complicated due to
the need to simulate many actions to reach the
initial state for a certain test case
46References
- A.F. Memon, GUI testing pitfalls and process,
Computer, v 35, n 8, Aug., pp. 87-88, 2002 - A.F. Memon and M.L. Soffa, Regression testing of
GUIs, Software Engineering Notes, v 28, n 5,
Sept., pp. 118-27, 2003 - R.K. Shehady and D.P. Siewiorek, A Method to
Automate User Interface Testing Using Variable
Finite State Machines, Proc. 27th Ann. Int'l
Symp. Fault-Tolerant Computing (FTCS '97), pp.
80-88, June 1997 - A.F. memon, M.E. Pollock, and M.L. Soffa,
Hierarchical GUI test case generation using
automated planning, IEEE Transaction on Software
Engineering, Vol. 27, Issue 2, Feb., pp. 144-55,
2001.
47References (Contd)
- D.J. Kasik and H.G. George, Toward Automatic
Generation of Novice User Test Scripts, Proc.
Conf. Human Factors in Computing Systems Common
Ground, M.J. Tauber, V. Bellotti, R. Jeffries,
J.D. Mackinlay, and J. Nielsen, eds., pp.
244-251, Apr. 1996 - Jacareto, http//jacareto.sourceforge.net/
- Abbot, http//abbot.sourceforge.net/doc/overview.s
htm - Selenium, http//www.openqa.org/selenium/.
- Marathon, http//www.marathon.com
- JFCUnit, http//jfcunit.sourceforge.net/
- MVC and MVP patterns http//www.martinfowler.com
48Thank you