A Comprehensive Framework for Testing Graphical User Interfaces

1
A Comprehensive Framework for Testing Graphical
User Interfaces

• Atif M. Memon
• atif_at_cs.pitt.edu
• Dept. of Computer Science
• University of Pittsburgh
• Advisors Mary Lou Soffa Martha E. Pollack

2
Research focus
50 of code
GUI
Interactions between the GUI and the underlying
code
Underlying Code
3
Challenges of GUI Testing

• What is the test input to the GUI?
• (Test case generation)
• How much testing is enough?
• (Test coverage)
• Is the GUI executing correctly during testing?
• (Test oracles)
• What test results can be salvaged from the
  previous test runs to test a new version?
• (Regression testing)
• How to represent the GUI to handle all the above?
• (Representation)

4
Current Techniques

• Capture/replay tools
• GUI test case generation
• FSM Esmelioglu and Apfelbaum and VFSM Shahady
  and Siewiorek models
• Genetic algorithm technique Kasik and George
• Visual TDE for GUIs Foster, Goradia, Ostrand,
  and Szermer

5
Goals of My Research

• Develop a comprehensive environment that
  integrates techniques (tools) to address various
  aspects of GUI testing
• Goals of developing the techniques
• Integrated for compatibility
• Automated avoid manual repetition of tasks
• Efficient to meet tight testing deadlines
• Robust to recognize a GUI error as soon as it
  occurs and report the problem
• Portable apply to multiple platforms
• General apply to a wide variety of GUIs

6
Framework

• New representation for GUIs
• Test case generation based on AI planning ICSE
  1999, AIPS 2000, IEEE TSE Feb 2001
• New event-based coverage criteria
• FSE 2001
• Unique test oracles FSE 2000
• Repairing test cases for automated regression
  testing
• Automated test executor QW 2000
• A comprehensive framework that integrates all the
  above components

7
Overview of the Framework
GUI Implementation Tools (Languages/Toolkits)
GUI Specifications
GUI Representation
GUI Implementer
Executing GUI
Regression Tester
Test Oracle
Test Case Generator
Test Coverage Evaluator
Test Executor
8
GUI Representation
GUI Implementation Tools (Languages/Toolkits)
GUI Specifications
GUI Implementer
GUI Model
Test Designer
Executing GUI
Regression Tester
Test Oracle
Test Case Generator
Test Coverage Evaluator
Test Executor
9
Creating the GUI Model

• Modeling
• GUIs state in terms of objects their
  properties
• Events as state transducers
• GUIs hierarchical structure
• GUI components
• Event-flow graphs
• Integration tree
• Classifying events

10
Modeling the GUIs State

• A GUI at time T is modeled using
• Objects O o1, o2, o3, , om
• Properties P p1, p2, p3, , pl, where pi is
  an ni-ary (ni gt 1) Boolean relation of the form

True/False
Property(oj, oa, ob, ,ox, value)
Property Name
Optional value of Property
Object
GUIs state S p1, p2, p3,,pn
11
Example Modeling the GUIs State
Form1
Button1
Label1
Align(Label1, Caption(Label1, Color(Label1, Font(L
abel1,
alNone) Files of type) clBtnFace) (tfont))
12
All Properties of Button1
13
Determining Objects Properties

• Manual Examination
• Specifications (reduced set)
• GUI being tested
• Toolkit/language (complete set)
• All available properties

Now we know how to represent the GUIs state
14
GUI Events

• GUIs state is not static
• Events change the GUIs state
• Events E e1, e2, e3, , en, associated with a
  GUI are functions from one GUI state Si to
  another state Sj
• Notation Sj ei(Si)

15
Example An Event
Sj e(Si)
State Si
Background color is not yellow
w19
set-background-color (w19, yellow)
Event e
Background color is yellow
State Sj
16
Representing Events

• Infeasible to give exhaustive specifications of
  the state mapping for each event
• No set limit to the number of objects a GUI can
  contain at any point in time
• There can be infinitely many states of the GUI
• Model the GUI events using operators, which
  specify their preconditions and effects

17
Operators

• An Operator is a triple
• ltName, Precondition, Effectsgt
• Name identifies an event and its parameters
• Precondition is a set of positive literals
• Effects is a set of positive or negative literals
• Operator Op is applicable in any state Si in
  which
• All the literals in Precondition(Op) are TRUE
• The resulting state Sj is determined by using
  Effects(Op)
• All positive literals in Effects(Op), and
• All literals that were TRUE in Si, except
• Those that are negative in Effects(Op)

18
Operator Example
Operator set-background-color Parameters wX
window col color Precondition isCurrent(wX),
background-color(wX, oldColor), oldColor !
col. Effects background-color(wX, col).
set-background-color (w19, yellow)
Primitive Operator
Now we know how to represent the events of the GUI
19
Model GUI Hierarchically

• Hierarchy
• GUIs are designed as a hierarchy of components
• Hierarchical model makes testing intuitive and
  efficient

20
Modal Windows in GUIs
Main
21
Modal Windows in GUIs
Main
invokes
Print
22
Modal Windows in GUIs
Main
Print
Components
Properties
23
Integration Tree
Main
FileNew
FileOpen
Print
FormatFont
FileSave
PageSetup
ViewOptions
Properties

• Definition Integration tree is a triple ltN, R,
  Bgt
• N is the set of components in the GUI
• R ? N is a designated component called the Main
  component
• B is the set of directed edges showing the
  invokes relation between components, i.e., (Cx,
  Cy) ? B iff Cx invokes Cy.

24
Representing a Component
File Edit Help
Event-flow Graph
25
Event-flow Graph
File
Help
Edit
Open
Save
About
Contents
Cut
Copy
Paste
To File, Edit and Help
To File, Edit and Help

• Definition Event-flow graph is a 4-tuple ltV, E,
  B, Igt
• V is the set of vertices, representing events,
• E is the set of directed edges, showing the
  follows relationship,
• B is the set of events first available (shown in
  red),
• I is the set of events that invoke other
  components (dotted lines).

26
Classifying Events

• Classification
• A new classification of events aids in creating
  the hierarchical model of the GUI

• Opening modal windows
• Restricted-focus events
• Closing modal windows
• Termination events
• Opening modeless windows
• Unrestricted-focus events
• Opening menus
• Menu-open events
• Interacting with underlying software
• System-interaction events

27
GUI Modeling Steps

• From the GUIs specifications (formal/informal),
• Identify objects and properties
• Create operators for GUI events
• Using the event classification, identify GUI
  components, create event-flow graphs and
  integration tree

28
Test Case Generator
GUI Implementation Tools (Languages/Toolkits)
GUI Specifications
GUI Implementer
GUI Model
Test Designer
Executing GUI
Regression Tester
Test Oracle
Tasks
Test-Case Generator
Test Coverage Evaluator
Test Executor
Test Case
29
GUI Test Cases

• Individual user events
• Not enough !
• Sequences of user events lead to different states
• Test case contains sequence of user events

30
Definition GUI Test Case

• legal event sequence
• e1e2e3 en is a legal event sequence
• if (ei, ei1) is an edge in an event-flow graph
• or ei invokes component Cx and ei1 is an event
  in Cx
• A GUI test case is a triple
• (S0, e1e2e3en, S1 S2 Sn)
• S0 is a GUI state, and
• e1e2e3en is a legal event sequence
• Si ei(Si-1), 1 lt i lt n

31
A Test Case for WordPad
S0
Expected State Sequence in Test Oracles
32
Selecting Event Sequences

• Infinitely many
• Manual
• Expert chooses sequences
• Automated
• Randomly choose sequences
• Structural (event-flow graph integration tree)
• Automatically generate events for COMMONLY USED
  TASKS

Multiple Event Sequences
This is the text.
This is the text.
Goal State
Initial State
33
A Plan for a GUI Task
Initial State
Goal State
This is the text.
This is the text.
34
Overview of Test Generation
35
Using Primitive Operators

• One operator for each event

Operator set-background-color Parameters wX
window col color Precondition isCurrent(wX),
background-color(wX, oldColor), oldColor !
col. Effects background-color(wX, col).
set-background-color (w19, yellow)
36
Exploit the GUIs Structure

• Reduce the number of operators
• System more efficient
• Easier for the test designer

37
Operator Abstractions

• Two types of abstractions
• Combine buttons ? create system-interaction
  operators
• Decompose GUI hierarchically ? create component
  operators
• Key idea the user interacts with one component
  at a time.

38
Create System-Interaction Operators
Sys-Interaction Operator
File_SendTo_MailRecipient ltFile SendTo
MailRecipient gt
39
Create Component Operators
Set Language
English (United States)
Using Primitive Operators Only
Default...
OK
Cancel
Main GUIs Operator Set
Using Abstraction
Set Language SelectFromList() Default OK Cancel
...
Language Windows Operator Set
Main GUIs Operator Set
SelectFromList() Default OK Cancel
Set Language ...
40
Using Component Operators
Language Windows Operator Set
SelectFromList() Default OK Cancel
High Level Plan
SetLanguage()
...
...
Planner
SelectFromList (English(US))
OK
Sub Plan
Define Abstraction
SetLanguage()
Component Operator
41
Effects of Exploiting the GUIs Structure

• Reduction in planning operators
• 362 operators ? 32 operators
• Ratio 101 for MS Wordpad
• 201 for MS Word
• Efficiency

42
Initial State
This is the text.
Goal State
This is the text.
43
Test Case
Primitive Operator
Component Operator
Component Operator
Primitive Operator
INITIAL
GOAL
FormatFont (This, 18pt)
FormatFont (text, Underline)
SelectText (text)
SelectText (This)
44
Different from HTN Planning
Primitive Operator
Component Operator
Component Operator
Primitive Operator
INITIAL
GOAL
SelectText (This)
FormatFont (This, 18pt)
FormatFont (text, Underline)
SelectText (text)
Planner
Planner
FormatFont
Underline
OK
FormatFont
18
OK
No Interactions
45
Alternative Test Case
Primitive Operator
Component Operator
Component Operator
Primitive Operator
INITIAL
GOAL
FormatFont (This, 18pt)
FormatFont (text, Underline)
SelectText (text)
SelectText (This)
Primitive Operator
Component Operator
Primitive Operator
Primitive Operator
SelectText (This)
SelectFromList (18)
FormatFont (text, Underline)
SelectText (text)
SelectFromList (18)
SelectText (This)
SelectText (text)
Format
Font
Underline
OK
46
Experiments

• Purpose
• To determine whether planning is a feasible
  approach for GUI test case generation
• Execution time
• Human effort
• Experimental design
• GUI our version of MS Wordpad (36 modal windows,
  362 events)
• Tasks 50 tasks (initial goal states)
• Test cases generated 290 test cases (6-56
  events) using the IPP planner
• Hardware platform 350 MHz Pentium based machine,
  256 MB RAM

47
Test Case Generation
48
Automated Execution
49
Test Oracles
GUI Implementation Tools (Languages/Toolkits)
GUI Specifications
GUI Implementer
GUI Model
Test Designer
Executing GUI
Oracle
Regression Tester
GUI State (run-time)
Expected-state Generator
Tasks
Expected-state sequence
Test-Case Generator
Execution Monitor
Test Coverage Evaluator
Verifier
Test Case
Actual State Information
Verdict
Test Executor
50
Determine Correct Behavior

• To check the GUIs state after each event
• Approaches
• Manual
• Automated

• Challenges
• Generating expected state
• Extracting actual state
• Comparing expected actual states

51
Overview of GUI Oracle
Event Sequence
Run-time information from executing GUI
Expected-state Generator
GUI Model
Expected State
Execution Monitor
Actual State
Verifier
Oracle
Verdict
52
Deriving Expected State

• Obtain S1 e1(S0)
• And Si ei(Si-1)

en
e1
e2
e3
S0
53
Obtaining Actual GUIs State

• Execution monitor
• Compatible with expected state
• Returns property(objects, value)
• e.g., Caption(button1, Cancel)
• Actual state can be obtained by
• Screen scraping
• Queries

54
Automated Execution
Event Sequence
Expected State
Level of checking
55
Experiments

• Purpose determine
• Time to derive expected state
• Time to execute monitor and verifier
• Experimental design
• GUI our version of MS Wordpad (36 modal windows,
  362 events)
• Test cases generated 290 test cases (6-56
  events) using an AI planner
• Hardware platform 350 MHz Pentium based machine,
  256 MB RAM
• Properties reduced set
• Level of checking GUI relevant property set

56
Deriving Expected State
Total CPU time (all 290 test cases and expected
states) 75.84 sec.
57
Automated Execution
GUI Relevant-properties checking Total running
time lt 10 minutes
58
Putting it All Together
GUI Implementation Tools (Languages/Toolkits)
GUI Specifications
GUI Specifications
GUI Implementer
GUI Model
Test Designer
Executing GUI
Oracle
GUI State (run-time)
Regression Tester
Expected-state Generator
Tasks
Expected-state sequence
Test-Case Generator
Coverage Criteria
Coverage Evaluator
Execution Monitor
Verifier
Test Case
Coverage Report
Actual State Information
Verdict
Test Executor
59
Future Work

• Extensions
• Web testing
• Interface Underlying code
• Relationship
• Separation
• GUI Specifications Testing
• Prioritizing GUI test cases
• New Directions
• Object-oriented Component-based
• Reactive
• Execution Profiles