An Overview of Modern Testing Tools and Methodologies

1
An Overview of Modern Testing Tools and
Methodologies

• Professor Ron Kenett
• Tel Aviv Unversity
• School of Engineering

2
Automated Software Testing Market

• Maturity
• New
• Undeveloped

• Players
• 3 main vendors
• Mercury, Segue and Rational
• Multiple niche vendors

• Strong Growth
• - CAGR of 41 between 1998-2002 (IDC)
• - Expected to exceed 1.2 billion by 2002

3
( K)
4
Automated Software Testing
State of Practice

• An error
• Costs
• 20K to 40K
• Effects
• Lost Revenues ...
• Operations ...
• Decision Making
  ...
• ...
  Clients !!!

5
Automated Software Testing State of Practice (2)

• Example
• Airline Reservation System (American/United)
• 1-minute DOWNTIME 20,000 !
• 10-minute coffee break 100,000 !
  (for the person who finds and fixes
  the error )

6
Automated Software Testing State of Practice (3)

• System Quality has become
• Companies ask

X
Past
Today
7
Automated Software Testing State of Practice (4)

• Relative cost of a BUG if detected at
• Hence The later S/W errors are found -
  
• - the more
  they cost !!!

Production Stage
270X
8
60
9
Market Trends New Software
Technologies

• WEB (e.g. Browser-based clients)
• Paradigms (e.g. Java, Objects, components)
• Infrastructures (e.g Corba)

10
Market Trends Impact on
applications and users

• Applications
• Reside on multiple servers
• Developed by multiple vendors
• Users
• No longer companys well-trained customers but
  rather WWW population - may range to millions !
• No direct support (for WEB-based applications)

11
Software Life Cycle With Dependencies
Requirements Analysis
Specification
Coding
System Test
Integration Test
Module Test
How information from earlier stages
can be used in the testing stages
12
Automated Software Testing Testing
Categories trends

• Each category
• Is complimentary to the other categories
• Focuses on an earlier stage of the products life
  cycle.

13
X/WinRunner
In-house tools
14
Specification-based Testing

• A James Martin study
• Poor/Missing Specifications ...
  
• 50 of the BUGS
  discovered in S/W !
• Erroneous specifications
• 80 of the money spent
  to fix BUGS !

To improve S/W testing, we need to improve the
process in which specifications are developed
and functional tests are designed
15
Specification-based Testing (2)

• Focus on
• A mechanism to derive tests from a specification
• Use the tests to assess the functional
  correctness of the program with respect to the
  specification.
• Status
• Still in its infancy only a few tools are
  available, most of which only produce test cases.

16
Automated Software testing Current Situation
Test Cases
Specification
Manually derived
Test Plan
Program

• No standards
• No formalism
• No commercial tools

17
The classical testing workflow
Requirements
No standards No formalism No tools
Manually derived
Specifications
Test Plan
Test Cases
S/W Design
Program exercise
Implementation
18
An alternative testing workflow
Requirements
Derived
Attributes
Specifications
Test Plan
Test Cases
Inputs Outputs
Automate Tests
S/W Design
Implementation
19
Automated Testing Tools Categories

• Most testing tools deal with the deployment
  stages of the product life-cycle. They can be
  categorized as
• Capture-replay
• Load/Performance
• Coverage
• Static Analysis
• Test Drivers

20
Examples of Automated Testing Tools

• TestMaster (Teradyne)
• a model-based testing tool that generates test
  cases from a model and links to various test
  execution environments
• SoftTest (Bender Associates)
• a case tool (based on requirement-based testing)
  - that constructs test cases from cause-effect
  graphs.

21
Automatic testing Tools Mapping
Capture/Replay
TestWEB
WinRunner
Load and Performance
CYRANO
SQA Suite LoadTest
Load Runner
preVue
Non-GUI Drivers Test Managers
TestExpert
TestWorks
Cantata
Test Evaluation
T-Scope
Panorama C/C
PureCoverage
Insure
Static Analysis
TestBed
Plum Hall SQS
PC-Metric
Hindsight/SQA
Specfication-based tools
SoftTest
TestMaster
22
(No Transcript)
23
The Internet Business Model
Corporation

• E-Commerce
• E-Service

24
Web Application Audience
Traditional
Users
Months
10s, 100s, ...
Client/Server
0 1 2 3 4 5
Users
Nowadays
Minutes
WEB Application
10Ks, 100Ks ...
0 1 2 3 4 5
25
Web Application Development
26
Multiple changing technologies
Active-X
XML
HTML
CORBA
Java
ASP
ODBC
VBScript
JavaScript
27
Complex user interface
28
Rapid development by non-professionals
Purchase
Purchase
Compare prices
Register
Purchase
Development
29
Testing Evolving Priorities
Show it Works Find Defects Measure
Quality Design Quality (Demonstration)
(Detection) (Prediction)
(Prevention) 1960s view
1970s view 1980s view
1990s view