Finite State ModelBased Testing on a Shoestring

1
Finite State Model-Based Testing on a Shoestring
Presented By Revital Eres
2
What are we going to talk about?

• Model-based testing technique
• Using The Visual Test tool (Rational Software
  ) we will
• 1. Create a finite state model of an application.
• 2. Generate sequences of tests actions from the
  model.
• 3. Execute the test actions against the
  application.
• 4. Determine if the application worked right.
• 5. Find bugs.

3
Running Example the Clock application of
Windows NT
Possible test Setup Put
the Clock into its Analog display mode   Action
Click on Settings\Digital  
Outcome Does the Clock correctly change to
the Digital display?  
4
What is a model?

• A model is an description of a system behavior.
• Models are simpler than the system they describe
  and they can help us understand and predict the
  systems behavior.

5
Motivation for using Model-based testing technique

• The waterfall model of software engineering
• Requirements analysis
• Specification
• Design
• Development
• Testing
• Maintenance

Typically people with business process skills
Software engineers with detailed knowledge of
software method and practice
6
Motivation for using Model-based testing technique
7
How can a model be used?

• Communication
• Generate code
• Generate test cases
• Provide an oracle for test cases

8
More advantages of using model-based testing

• Maintenance easier to maintain Model than
  test cases.
• Can generate more test cases.

9
Sample problem - File system
Open_for Read Open_for_Write Close
File 3
Accept_Command Reject_Command File Status
Files
Users
10
Step One create a Finite State Model of an
Application

• The Operation modes
• System mode
• NOT_RUNNING means Clock is not running
• RUNNING means Clock is running
• Setting mode
• ANALOG means Analog display is set
• DIGITAL means Digital display is set

11
Step One create a Finite State Model of an
Application

• The Actions
• Start the Clock application
• Stop the Clock application
• Select Analog setting
• Select Digital setting

12
Step One create a Finite State Model of an
Application
The Rules Start o        If the
application is NOT running, the user can execute
the Start command. o        If the application is
running, the user cannot execute the Start
command. o        After the Start command
executes, the application is running.     Analog o
        If the application is NOT running, the
user cannot execute the Analog command. o       
If the application is running, the user can
execute the Analog command. o        After the
Analog command executes, the application is in
Analog display mode.  
13
Step One create a Finite State Model of an
Application
The Rules Digital o        If the
application is NOT running, the user cannot
execute the Digital command. o        If the
application is running, the user can execute the
Digital command. o        After the Digital
command executes, the application is in Digital
display mode. Stop o        If the application
is NOT running, the user cannot execute the Stop
command. o        If the application is running,
the user can execute the Stop command. o       
After the Stop command executes, the application
is not running.  
14
Step One create a Finite State Model of an
Application
for system_mode NOT_RUNNING to RUNNING ' for
all system modes for setting_mode ANALOG to
DIGITAL ' for all setting modes for action
Start to Digital ' actions Start,Stop,Analog,Dig
ital
for system_mode NOT_RUNNING to
RUNNING ' for all system modes for
setting_mode ANALOG to DIGITAL ' for all
setting modes for action Start to
Digital ' actions Start,Stop,Analog,Digital
possible TRUE ' assume action is
possible new_system_mode system_mode '
assume mode values do not change new_setting_mo
de setting_mode select case action case
"Start" 'start the clock
if (system_mode NOT_RUNNING) then
'clock must be NOT running
new_system_mode RUNNING
'clock goes to running else possible
FALSE endif case "Stop"
'stop the clock if (system_mode
RUNNING) then 'clock must be
running new_system_mode
NOT_RUNNING 'clock goes to NOT
running else possible
FALSE endif
case "Start" 'start the clock
if (system_mode NOT_RUNNING) then
'clock must be NOT running new_system_mode
RUNNING 'clock goes to
running else possible FALSE endif
15
Step One create a Finite State Model of an
Application
case "Analog"
'choose analog mode if
(system_mode RUNNING) then 'clock
must be running new_setting_mode ANALOG
'clock goes to analog
mode else possible FALSE endif
case "Digital" 'choose digital
mode if (system_mode RUNNING) then
'clock must be running new_setting_mode
DIGITAL 'clock goes to digital
mode else possible FALSE endif
end select if (possible TRUE) then ' if
action possible print system_mode"."setting_m
ode, ' print begin state print action, '
print action print new_system_mode"."new_sett
ing_mode ' print end state endif next
action next setting_mode
next system_mode
if (possible TRUE) then ' if action
possible print system_mode"."setting_mode, '
print begin state print action, ' print
action print new_system_mode"."new_setting_mod
e ' print end state endif
16
Step One create a Finite State Model of an
Application
The state table
17
State Transition Diagram for the clock Model
18
Step Two - Generate sequences of tests actions
from the model

• Testing an application is like following a path
  through a finite state model.
• Different types of traversals can meet different
  needs of testers.
• One of the following strategies can be adopted
• - Execute every possible action.
• - Testing combinations of Actions.
• - Random Walks.
• - Testing the most likely paths.

19
Step Two - Generate sequences of tests actions
from the model (Cont.)

• - Execute every possible action

20
Step Two - Generate sequences of tests actions
from the model (Cont.)

• Graph theory techniques allow us to use the
  behavioral information stored in models to
  generate new and useful tests.
• Euler cycle - a graph cycle which uses each edge
  exactly once.
• A connected graph has an Eulerian circuit iff it
  has no vertices of odd degree.

21
Step Two - Generate sequences of tests actions
from the model

• The Chinese Postman Problem
• Given that it is impossible to cross each bridge
  exactly once and return to the starting point,
  what is the minimal amount of re-crossing a
  walker needs to do? (Kwan 1962)

22
Step Two - Generate sequences of tests actions
from the model

• The Chinese Postman Solution
• Eulerizing the graph by duplicating links and
  performing an Euler tour.

a b c b e f g d d
23
Step Two - Generate sequences of tests actions
from the model
New York Street Sweeper Problem What is the
minimal length tour when the links only allow
passage in one direction? (Beltrami 1977, Bodin
Tucker 1983)

24
Step Two - Generate sequences of tests actions
from the model
New York Street Sweeper Solution Eulerizing
the graph by duplicating links and performing an
Euler tour.

a b c b f e g d e g
25
Step Two - Generate sequences of tests actions
from the model
Back to our original mission Execute every
possible action Eulerizing the graph by
duplicating links and performing an Euler
tour.

a b c b f e g d e g
26
Step Two - Generate sequences of tests actions
from the model
Testing combinations of Actions. The problem -
Find a path which contains every combinations of
2 arcs in the graph, for example a path which
contains bc, bf and bg

a b c b f e g d e g
27
Step Two - Generate sequences of tests actions
from the model
Testing combinations of Actions. The
solution de Bruijn algorithm- Creating a dual
graph

a b c b f e c b g d e f e g
28
Step Two - Generate sequences of tests actions
from the model

• Create a dual graph of the original graph (i.e.,
  a graph in which the arcs of the original graph
  are converted to nodes)
• Everywhere in the original graph that arc 1 is an
  incoming to a node and arc 2 is outgoing from the
  same node, create an arc in the dual graph from
  node 1 to node 2. For instance, in the left hand
  graph in Figure 6, arc a is incoming to the
  node from which arc b is outgoing therefore in
  the dual graph there is an arc from node a to
  node b.
• Eulerize the dual graph (by duplicating arcs to
  balance node polarities)
• Traverse the Eulerized graph, noting the names of
  the nodes that you pass.

29
Step Two - Generate sequences of tests actions
from the model

• Random Walks From the current node, choose an
  outgoing link at random, follow that link to the
  next node and repeat the process.
• Advantages
• - Simple to implement.
• - Can be very useful.
• Disadvantage
• - Could be very inefficient about covering a
  large graph quickly.

30
Step Two - Generate sequences of tests actions
from the model

• Testing the most likely paths
• Markov Model- a way to bias random walk toward
  the activities that a user is more likely to
  perform.

31
Step Two - Generate sequences of tests actions
from the model

• Assign probabilities to the links in the graph
  (as with regular Markov chains)
• Establish a minimum probability for the paths
  that you are interested in, such as 0.0000001
• Starting at the initial node, move through the
  graph, keeping track of the cumulative
  probability of the links youve traversed so far.
• Continue traversing links in the graph until you
  reach the final node of the graph or until the
  cumulative probability of the path falls below
  the minimum youve established.
• If you have reached the final node, record the
  path that got you there as well as its
  probability.
• If you have merely fallen below the minimum
  probability, back up and try a different link.
• Once you have collected all the paths in the
  graph that have cumulative probability greater
  than your minimum, sort them in order of their
  cumulative probability, from highest to lowest.

32
Step Two - Generate sequences of tests actions
from the model

• Chinese Postman tour on our example
• Start
• Analog
• Digital
• Digital
• Stop
• Start
• Analog
• Stop
•  

33
Step Three - Execute the test actions against the
application

• Take advantage of the functions Visual Test can
  offer to interact with the application we are
  testing

34
Step Three - Execute the test actions against the
application
open "test_sequence.txt" for input as
infile get the list of actions  while not
(EOF(infile))   line input infile,
action read in an action   select case
action  case "Start" start the
Clock run("C\WINNT\System32\clock.exe) VT
call to start Clock program Upgraded   case
"Analog" choose analog mode WMenuSelect("Se
ttings\Analog") VT call to select menu item
Analog   case "Digital" choose
digital mode WMenuSelect("Settings\Digital") V
T call to select menu item Digital   ca
se "Stop" stop the Clock WSysMenu
(0) VT call to bring up systemmenu WMenuSele
ct ("Close") VT call to select menu
item Close   End select   Test_oracle()
determine if Clock behaved correctly  
end  
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
Step Four - Determine if the application worked
right.

• Using an Oracle we determine if the output we
  observed in the end of the test was correct.
• One of the benefits of model-based testing is the
  ability to create a test oracle from the state
  model.

39
Step Four - Determine if the application worked
right.
WFndWnd(Clock)
Determine if a Clock is running
Determine if the Analog or Digital display is
showing
40
Step Four - Determine if the application worked
right.
if (system_mode RUNNING) then if (
WFndWnd("Clock") 0 ) then 'if no "Clock"
running print "Error Clock should be
Running" 'print the error stop endif if (
(setting_mode ANALOG) _ 'if analog mode AND
NOT WMenuChecked("Settings\Analog") ) then 'but
no check next to Analog print "Error Clock
should be Analog mode" 'print the
error stop elseif ( (setting_mode DIGITAL)
_ 'if digital mode AND NOT WMenuChecked("Settin
gs\Digital") ) then 'but no check next to
Digital print "Error Clock should be Digital
mode" 'print the error stop endif endif
41
Step Five - Find bugs
42
Step Five - Find bugs
43
What did we learn?

• Using a running example we implemented the
  Model-based testing technique which demonstrated
  the fact that it is an efficient technique for
  generating test cases based on the application
  behavior.
• We saw different types of tests that can be
  generated automatically from traversing the
  model.
• Unlike traditional testing this technique enables
  us to maintain, review and update the test
  programs more easily.

44
References

• 1. Finite State Model-Based Testing on a
  Shoestring
• Harry Robinson
• 2. Graph Theory Techniques in Model-Based
  Testing
• Harry Robinson
• 3. A methodology and architecture for automated
  software testing
• Gronau, Hartman
• http//www.geocities.com/model_based_testing//onli
  ne_papers.htm