Texas A&M University Department of Computer Science CPSC

1
Model Based Testing

• Texas AM University
• Department of Computer Science
• CPSC 606 SOFTWARE ENGINEERING
• Presentation 2
• Sameh S. Sharkawi

2
Outline

• Introduction
• What is MBT
• Motivation behind MBT
• How MBT Works
• Benefits of MBT
• Problems of MBT
• Current State of MBT in Industry
• Software Tools for MBT
• Conclusion
• References

3
Introduction (What is MBT)

• Definition
• Model-based testing refers to software testing
  where test cases are derived in whole or in part
  from a model that describes some (usually
  functional) aspects of the system under test
  (SUT) .
• Wikipedia Definition of Model Based Testing.

4
Introduction (What is MBT Contd.)

• MBT is an approach in which you define the
  behavior of a system in terms of actions that
  change the state of the system.
• Such a model of the system results in a
  well-defined Finite State Machine (FSM) which
  helps us to understand and predict the systems
  behavior

5
Introduction (Motivation behind MBT)

• Testing is expensive
• 30-50 of development costs
• Poor testing is VERY expensive
• Down time
• Maintenance costs
• Rework
• Law suits
• Model Based Testing promises
• Increased effectiveness of testing
• Similar or decreased costs
• Reuse of design artifacts

6
Introduction (Motivation behind MBT Downtime
Costs)

• Brokerage operations 6,450,000
• Credit card authorization 2,600,000
• Ebay (1 outage 22 hours) 225,000
• Amazon.com 180,000
• Package shipping services 150,000
• Home shopping channel 113,000
• Catalog sales center 90,000
• Airline reservation center 89,000
• Cellular service activation 41,000
• On-line network fees 25,000
• ATM service fees 14,000

7
How MBT Works
8
How MBT Works?

• The model is usually an abstract, partial
  presentation of the system under test's desired
  behavior.
• The test cases derived from this model are
  functional tests on the same level of abstraction
  as the model.
• These test cases are collectively known as the
  abstract test suite

9
How MBT Works? Contd.

• The abstract test suite cannot be directly
  executed against the system under test because it
  is on the wrong level of abstraction.
• Therefore an executable test suite must be
  derived from the abstract test suite that can
  communicate with the system under test.
• This is done by mapping the abstract test cases
  to concrete test cases suitable for execution.

10
How MBT Works? Contd.

• Deriving Test Cases Algorithmically
• Test Case Generation by Theorem Proving
• Test Case Generation by Constraint Logic
  Programming
• Test Case Generation by Model Checking
• Test Case Generation by Symbolic Execution

11
How MBT Works? Contd.

• Spec Explorer (UW-MSR Summer Institute '04)

Model
Provides expected results for
Generates
PassFail
Test Oracle
Test Cases
Provides actual results for
Run
Implementation
12
Benefits of MBT
13
Benefits of MBT

• Starting from specification
• Involves testers early in the development process
• Teams testers with developers
• Forces testability into product design
• Building the test interface
• Finds design and specification bugs - before code
  exists
• The model is the test plan - and is easily
  maintained
• Automated test suite generation
• Coverage is guaranteed - increases testing
  thoroughness
• Zero test suite maintenance costs
• Automated test suite execution
• Finds code and interface bugs
• Includes a framework for the testing of
  distributed applications
• Reduces test execution costs

14
Benefits of MBT Contd.

• These benefits can be summarized as
• Machine
• Time

15
Benefits of MBT Contd.

• Machines
• A typical test engineer
• Earns 41 500 (50 000 USD) per year
• Works 40 to 50 hours per week
• A typical test machine
• Costs 250 (300 USD) per year to buy and operate
• Can work 100 hours per week

16
Benefits of MBT Contd.
react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf
17
Problems of MBT
18
Problems of MBT
19
Problems of MBT Contd.
20
Problems of MBT Contd.
21
Problems of MBT Contd.
22
Problems of MBT Contd.

• Process shift
• Up front investment in test
• Personnel shift
• Higher education and sophistication

Spec Explorer (UW-MSR Summer Institute '04)
23
Current State of MBT in Industry
24
Current State of MBT in Industry

• IBM Research
• Most projects were done by PhD Holders
• Very Little was successful (Just in the Lab,
  never actually went into market)
• Too Complicated to use and analyze
• Microsoft Research
• The Spec programming system is a new attempt at
  a more cost effective way to develop and maintain
  high-quality software

25
Current State of MBT in Industry

• Spec is a computer language which includes and
  extends C
• Adds pre/post conditions, contracts
• Adds high-level data types with convenient
  notations
• Adds logical quantifiers like FORALL and EXISTS
• A Spec model is just a program!
• You can run it like a C program
• It can call framework code
• NEW You can explore it (run all of its possible
  behavior) which is how we do MBT with Spec

Spec Explorer (UW-MSR Summer Institute '04)
26
Current State of MBT in Industry (Spec Explorer)

• A model exploration and testing tool for .NET
• Authoring of models in Word
• Model exploration (generating FSM)
• FSM visualization
• FSM traversal and test suite generation
• Automatic implementation binding
• Online test-suite execution and offline
  test-suite code generation

Spec Explorer (UW-MSR Summer Institute '04)
27
Current State of MBT in Industry (Spec )

• Model Calculator (viewpoint only starting and
  stopping, and switching scientific mode)

bool Running false bool Scientific
false Action void SetRunning(bool
newRunning) requires Running ! newRunning
Running newRunning Action void
SetScientific(bool newScientific) requires
Running requires Scientific ! newScientific
Scientific newScientific
System State
Spec Pre-Condition describes when action is
enabled
State Update
Tells Spec Explorerthis is an action. Action
invocations do appear in tests.
Spec Explorer (UW-MSR Summer Institute '04)
28
Software Tools for MBT
29
Software Tools for MBT

• Conformiq Test Generator is a model-based testing
  tool using UML state charts that represent
  testing strategies
• Leirios Test Generator is a model-based testing
  tool that generates tests automatically from
  deterministic system specifications
• Reactis Tester is another model-based testing
  tool that focuses on control systems
• TGV is a tool for the generation of conformance
  test suites for protocols
• TorX is also a prototype testing tool for
  conformance testing of reactive software
• Lurette is an automated testing tool of reactive
  programs written in Lustre
• AsmL Test Tool can generate tests directly from
  an AsmL model
• AutoFocus (in german) is a graphical tool for
  developing and modeling distributed systems with
  integrated testing facilities

30
Software Tools for MBT (AsmL)

• AsmL is the Abstract State Machine Language. It
  is an executable specification language based on
  the theory of Abstract State Machines. The
  current version, AsmL 2 (AsmL for Microsoft
  .NET), is embedded into Microsoft Word and
  Microsoft Visual Studio.NET. It uses XML and Word
  for literate specifications. It is fully
  interoperable with other .NET languages. AsmL
  generates .NET assemblies which can either be
  executed from the command line, linked with other
  .NET assemblies, or packaged as COM components.

31
Conclusion
32
Conclusion

• Modeling is not easy
• What details do you leave out?
• How do you check the model?
• There are no recipes for success
• What needs to happen
• Easier notation
• Incremental approach
• Pilots and examples

33
Questions

• How do we guarantee that the generated expected
  outputs and inputs are enough and cover all the
  cases?
• How to make for the errors and exceptions that
  may occur due to running the program on different
  environments and different OSs?
• How is automated generation of inputs and outputs
  work when an actual human interaction is required
  as part of the system testing?
• Does MBT cover stress testing?
• Does MBT cover how user friendly the system is or
  it only covers functionality?

34
References
35
References

• http//en.wikipedia.org/wiki/Model-based_testing
• http//blogs.msdn.com/nihitk/articles/144664.aspx
• http//research.microsoft.com/specsharp/
• http//react.cs.uni-sb.de/mbt2006/talks/mbt4masses
  .pdf
• http//react.cs.uni-sb.de/mbt2006/talks/ModelBased
  TestingSoberEvaluation.pdf
• http//aetgweb.argreenhouse.com/papers/1999-icse.p
  df
• Colin Campbell, Wolfgang Grieskamp, Lev
  Nachmanson, Wolfram Schulte, Nikolai Tillmann,
  Margus Veanes Spec ExplorerAn Integrated
  Environment for Model-Based Testing. Foundations
  of Software Engineering Microsoft Research,
  Redmond