Automated Software Testing using Open Source Testing Tools

1
Automated Software Testing using Open Source
Testing Tools By Elfriede Dustin
2
Agenda

• Who is IDT
• State of Software Testing
• What is Automated Test and Re-test (ATRT)
• Approach to ATRT
• Advantage of open source
• ATRT and open source Our IDE and solution
• ATRT Challenges

3
Who is IDT?

• IDT specializes in the design, development, and
  implementation of Automated Software Testing and
  Re-Test (ATRT) and Quality Assurance (QA)
  solutions
• Deliver turn key automated test suite
• Automated test strategy and identification of
  highest payoff areas to apply automation for your
  project
• Selection of best automation tools for your
  project
• Using your existing test cases or develop test
  cases for you
• Test results documented in requirements
  traceability matrix
• Training and pilot project implementation with
  companies

4
Who is IDT?

• IDT specializes in the design, development, and
  implementation of Automated Software Testing and
  Re-Test (ATRT) and Quality Assurance (QA)
  solutions
• Deliver turn key automated test suite
• Automated test strategy and identification of
  highest payoff areas to apply automation for your
  project
• Selection of best automation tools for your
  project
• Using your existing test cases or develop test
  cases for you
• Test results documented in requirements
  traceability matrix
• Training and pilot project implementation with
  companies

• We are currently hiring Java Developers
• Plus wed like to hire 2 or 3 grad students to
  work for us part time ( i.e. 20 hours week )
• must be US citizen

5
State of Software Testing - Why ATRT?
?
6
State of Software Testing - Why ATRT?
Utilize Automated Testing Strategies and
Technology to Improve Productivity and Quality
50 or More of Overall Development Cost is
Typically Spent on Testing
Spiral / Release 4
Reduction in Test Days
Spiral / Release 3
Increased Test Coverage
Size Complexity of SW Baseline
Test Days
Spiral / Release 2
Manual
Test Days
Spiral / Release 1
Automated
Test Days
Manual vs. Automated
Test Days
IDT Provides Automated Software Testing Solutions
7
What is ATRT?

• Our definition of ATRT is
• Application and implementation of software
  technology throughout the entire Software Testing
  Life Cycle (STL) and QA lifecycle with the goal
  to improve STL efficiencies and effectiveness

8
ATRT Spans the Software DevelopmentLife Cycle
Visual Modeling
Design
Build
Assemble
Round-Trip Engineering
Test
Use cases
9
ATRT Spans the Software DevelopmentLife Cycle
Requirements
10
ATRT Spans the Software DevelopmentLife Cycle
Automated Test Tools
Visual Modeling
Development
Management
Execution
Automated Testing Tools
Automated Software Testing
11
ATRT Spans the Software DevelopmentLife Cycle
Integrated Suite of Tools
Requirements Management and Process Automation

• Additionally
• Middleware
• Infrastructure
• Defect Tracking
• Configuration Management
• Memory Leak Detectors
• Performance Testing Tools
• Documentation Tools
• others

Visual Modeling
Infrastructure
12
Types of Software Test Life-cycle Support Tools

• Which tools are you familiar with?

13
Types of Automated Test Tools
14
Types of Tools (Contd)
15
Types of Tools (Contd)
16
Deployment Approach Adapted ATLM

• Populate Test Manager with Test Cases and
  Requirements
• Develop Test Scripts from Test Procedures
• Test Results
• Pass/Fail by Test Case
• Pass/Fail Summary
• Test Time Required

• Identify or Develop
• Pilot
• Test Manager
• Test Scripts
• Data Comparators
• Performance Test Tools
• GUI Record/Playback
• Storing/Retrieving Results

• Collect/Create
• Requirements
• Test Cases
• Test Procedures
• Expected Results
• Interface Specifications
• System/Component
• Configuration Description

17
Why Open Source

• ?

18
Why Open Source?

• Advantage of Open Source
• no licensing issues i.e. license cost
  maintenance etc.
• easily modifiable
• adaptable
• lightweight
• flexibility - not tied to one vendor

19
ATRTs IDE - Eclipse

• Eclipse this site lists all the plugins
• http//www.eclipse-plugins.info/eclipse/plugins.j
  sp?categorySCMpager.offset0firstItem1
• Eclipse/TPTP Open Source - Developed by IBM
• Eclipse/Test and Performance Tools Platform
  (TPTP) offers a common extensible framework for
  the following functions
• Requirements Management http//sourceforge.net/pro
  jects/osrmt/
• Configuration Management - Subversion
• Testing Tools (http//www.nabble.com/Eclipse-TPTP-
  --Testing-Tools-f2262.html)
• Profiling default Java applications, but can be
  extended to other apps

20
ATRTs IDE - Eclipse

• Eclipse/Test and Performance Tools Platform
  (TPTP) offers a common extensible framework for
  the following functions (cont)
• Static Analysis default for C and Java, but
  can be extended to other apps
• Application monitoring and log analysis
• Build tool ant
• Xml input and output - using an open source Java
  package http//jakarta.apache.org/ecs/ , which is
  a Java library that has an API for constructing
  and outputting XML. 
• Test Manager STAF/STAX

21
Current Test Tool Implementation Example and
Challenges

• Vendor-provided (Capture/Playback) Tool
• Automated test tools mimic actions of the test
  engineer.
• During testing, the engineer uses the keyboard
  and mouse to perform some type of test or action.
• Testing tool captures all keystrokes and
  subsequent results, which are baselined in an
  automated test script.
• During test playback, scripts compare latest
  outputs with previous baseline.
• Testing tools have built-in, reusable test
  functions.
• Most test tools provide for non-intrusive
  testing i.e., they interact with the
  application-under-test as if the test tool was
  not involved.

22
Current Testing Tools Challenges (cont)

• In-House Software Development required
• Capture/Playback tools generate hard-coded
  values test scripts are not reusable, nor do
  they implement software development best
  practices right out of the box scripts need to
  be modified.
• Capture/Playback tools dont necessarily provide
  all testing features required code enhancements
  are often required to meet testing needs.
• Capture/Playback tools are not necessarily
  compatible with system engineering environment,
  and software testing scripts need to be developed
  in-house.
• Developed testing scripts are not cross-vendor
  compatible, even if the same scripting language
  is used

23
Linux Capture/Playback Tools IDT Findings
24
Automated Test and Re-Test (ATRT)

• ATRT includes a common framework comprised of
  re-usable software services
• Open Source Software Test Automation Framework
  (STAF)
• Provides automated test procedures, test inputs,
  and test results for SW components
• Suitable for use by 3rd parties not associated
  with development of the component

25
What is STAX?

• STAX is an automation system
• With STAX you can automate and monitor your
  entire test environment
• System/product setup
• Testcase distribution
• Testcase execution
• Testcase results collection and analysis
• STAX consists of
• A programming language designed for automation
• Programs written in this language are called jobs
• An execution engine (similar to an interpreter
  for other programming/scripting languages) which
  manages the execution and runtime behavior of the
  job
• A GUI application, called the STAX Monitor, which
  provides a dynamically updated view of your
  executing jobs
• A tool (STAXDoc) that is used to generate
  documentation for your STAX jobs

26
STAF can run on
    Windows 95    Windows 98 (and 98 SE)   
Windows Millenium Edition    Windows NT
Version 4.0    Windows 2000    Windows
XP    Windows Server 2003 (IA32, IA64,
AMD64)    Windows Vista (IA32, AMD64)   
Linux (Intel32/64, AMD64, PPC32/64)    Linux
on zSeries (31-bit, 64-bit)    AIX Version
4.3.3.0 or higher (32-bit, 64-bit)    Solaris
(Sparc) 2.6 and higher    Solaris AMD Opteron
64-bit    HP-UX 11.00 and higher (PA-RISC,
IA64 32-bit and 64-bit)    OS/400 V5R2 or
higher    z/OS UNIX V1.4 and higher
27
STAF Internal Services

• DIAG Provides diagnostics services Internal
  ("DIAG")
• DELAY Provides a means to sleep a specified
  amount of time Internal ("DELAY")
• ECHO Echos back a supplied message Internal
  ("ECHO")
• FILE SYSTEM Allows you to get and copy files
  across the network Internal ("FS")
• HANDLE Provides information about existing STAF
  handles Internal ("HANDLE")
• HELP Provides Help on STAF error codes Internal
  ("HELP")
• MISC Handles miscellaneous commands such as
  displaying the version of STAF
• that is currently running Internal ("MISC")
• PING Provides a simple is-alive message Internal
  ("PING")
• PROCESS Allows you to start, stop, and query
  processes Internal ("PROCESS")
• QUEUE Provides a network-enabled IPC mechanism
  for STAF Programs Internal ("QUEUE")
• SEMAPHORE Provides network-enabled named event
  and mutex semaphores Internal ("SEM")
• SERVICE Allows you to list services available on
  a machine and to examine the
• Requests that have been submitted on a
  machineInternal ("SERVICE")
• SHUTDOWN Provides a means to shutdown STAF and
  register for shutdown
• Notifications Internal ("SHUTDOWN")

28
STAF External Services

• The executable code for external STAF services
  resides outside of STAFProc, for example in a
  Java jar file, a C DLL file, or a Rexx script
  file.
• CRON Calls into STAF services at a specified time
  interval External (Java)
• EMAIL Allows you to send email messages External
  (Java)
• EVENT Provides a publish/subscribe notification
  system External (Java)
• EVENTMANAGER Allows you to call STAF services
  when a specified Event occurs External (Java)
• HTTP Allows you to make HTTP requests which can
  be grouped together in a session External (Java)
• LOG Provides a full-featured logging facility
  External (C)
• MONITOR Allows a testcase to publish its current
  running execution status for others to read
  External (C)
• RESOURCE POOL Allows you to manage exclusive
  access to pools of elements, e.g. VM UserIDs or
  Software Licenses External (C)
• STAX Provides an XML-based execution engine
  External (Java)
• ZIP Provides a means to zip/unzip/list/delete
  PKZip/WinZip compatible archives External (C)

29
Technologies used in STAX

• STAF
• STAF provides the infrastructure on which STAX
  builds. The full power of STAF and its services
  is exposed for use within STAX jobs.
• XML
• The STAX programming language is based on XML.
  This provides built-in structure to your jobs, as
  well as providing a set of existing tools for use
  in constructing your jobs, such as XML (aware)
  editors and XSLT.
• Python
• The STAX programming language builds on Python to
  provide a rich and accessible data model
• STAX's Python integration also allows you to
  access the wealth of existing Python libraries
• Java
• The STAX programming language allows you to
  access existing Java classes/libraries, providing
  another source for reuse

30
Automation Tasks
Automation Startup
System Setup
Execution
Monitoring
Testcase Execution
Synchronization
Testcase Output Analysis
Resource Management
Testcase Cleanup
Results Notification
Automation Completion
31
End-to-End Automation with STAF and STAX
Automation Startup
S T A X J o b
Event, EventManager, Cron
ltfunctiongt ltparallelgt ltsequencegt ltprocessgt ltstafcm
dgt lttestcasegt lttimergt ltblockgt ltloopgt ltmessagegt ltlo
ggt ltimportgt ltjobgt ltiterategt ltparalleliterategt lthol
dgt ltreleasegt ltterminategt
System Setup
FS, Process
Testcase Execution
Process, Monitor, Log, Variable, Queue, ResPool
Testcase Output Analysis
Log, Process
Testcase Cleanup
Process
Results Notification
Email, HTTP
Automation Completion
32
Key ATRT Architecture Requirements

• Support applications running on multiple
  computers
• Support applications developed in different
  languages
• Support applications running on different types
  of OSs
• Support applications which have GUI and those
  which do not (for example Interface testing)
• Support applications which use different types
  of network protocols such as TCP/IP, DDS, etc
• Support integration of multiple commercial
  testing tools from different vendors ( allowing
  as new or better products emerge in the market
  they can be utilized )
• Support testing w/o having to install ATRT on
  the same computers as the application under test
  and be able to be distributed across computers

33
Prototype Proof of Concept
Analysis Display Application
Display Application
Display Application
System Under Test
Windows Based
Simulation of Application Processing Messages
Eggplant R/P
VNC Robot R/P
Simulation of Application Processing Messages
ATRT
Test Manager RTM Bugzilla
Linux Based
STAF/STAX
34
ATRT Prototyping Effort

• Able to successfully execute ATRT for 48 hours
  without operator intervention
• During the tests
• VNCRobot - Approximately 25,000 operator key
  strokes were replayed
• STAF/STAX - More than 1,000,000 messages
  processed and verified by ATRT

35
Additional Requirement

• Pristine test environment required automated
  testing tool cannot be installed on
  Application-Under-Test PC
• Eggplant allows for remote capture/playback
  via VNCserver and KVM switch
• VNCRobot

36
Automated Software Testing

• Software testing paradox
• What is it?

37
Effective Automated Software Testing

• Current State
• Developing software in order to test software
• Automate the test support development automated
  IDL to code generation test data generation
  (using FireEye from NIST) compare utilities log
  file analyzers
• Our Automated Software Testing Goal
• Minimize Human interaction
• Possibly get on par with Hardware Automated
  Testing?

38
Hardware Testing current state

• Many automated test generators for manufacturing/
  logic devices, circuit testing, etc., available
  i.e. SEI/ATG, uMaster, etc.
• Automated testing without human interaction is
  the norm for hardware testing

39
Hardware vs. Software AT

• How is hardware testing different?
• Software Challenge
• Changing requirements
• GUI interfaces
• Code access/availability
• Defects uncovered earlier in the software
  development lifecycle are cheaper to fix
• Solution flexibility required problem dependent

40
Proposed solutions

• Focus on Reusable Components
• Test Data / Expected Results
• Interfaces
• Middleware
• Assuming code and models are available
• Model based testing automated test case
  generators based on models
• Eclipse effective ATRT IDE
• Probes see Aprobe Code instrumentation
• Building self-testable components
• Using standard language MOF to Text to generate
  automate test cases
• GUI Testing tool ideas
• New tools - other
• Working with NIST to produce additional tools

41
GUI Testing

• Jemmy - java library to test UI
• http//jemmy.netbeans.org/
• http//abbot.sourceforge.net/doc/overview.shtml
• Perl modules
• run via VNCServer?

42
Questions

• ?
• Email edustin_at_idtus.com with any follow-up
  questions

43
IDT

• We are hiring.
• Please come see me now if interested or send
• email edustin_at_idtus.com with any follow-up