Sonam Chauhan

1
webMethods Testing at Corporate Express

• Sonam Chauhan
• Corporate Express
• sonam.chauhan_at_ce.com.au

2
Overview

• Introduction
• Change Process
• General Requirements for Testing
• Old Testing Process
• New Testing Process
• Implementation Tips
• Future Steps and Trends
• References

3
Introduction

• Constant pressure for change
• Changes need to be tested for bugs
• Testing has business impact
• Good testing costs money
• Bad testing costs more money!
• Need to find the Golden Mean

4
Introduction

• webMethods issues that relate to testing
• Dev/Test/UAT/Live Builds ? Much Variation
• Due to ISs multiple packages feature
• Source Code Management ? Rudimentary
• Support planned for interfacing to SCMs
• Complex Change Deployments ? Risky
• Partially addressed by wMDeployer
• Testing Support ? Absent
• wM arent adding it

5
Change Process
Requirements Specification
Requirements Definition
Changes
ValidatedChanges
Review and Testing
Change Deployment (Dev ?Test ? UAT ? Live)
6
General Requirements for Testing

• Like development, testing is an engineering skill
• Allocate limited resources for max. benefit.
• Black-box Integration testing for Regressions ?
  Best return for your testing buck
• High level tests simulate live use
• Tests can be external to code
• Easy to retrofit to existing codebase
• Regression are the costliest bugs
• All changes must first do no harm
• This is not Unit Testing
• Unit tests cant assure system functionality
• This is integration/system-level testing

7
General Requirements for Testing

• Developers can be testers!
• Commitment to quality
• Make test development part of development process
• Make reviews of the test code part of code review
  process
• Automate testing
• Automated Regression Testing Harness
• Without automation, testing is tedious work
• Automation increases productivity and efficiency
• Permanently add hard-won insight to corporate
  memory
• And replay it on demand
• Essential for managing change in complex systems
• Active testing catches problems early
• Load testing often very important
• Some bugs can only be "teased out" under load.

8
Old Testing Process

• Implemented Circa 2001
• Mostly manual
• Repository of test cases
• Test case input submitted to system being tested
• System output compared to stored test case
  outputs and Diff generated (image at right)
• Diff manually inspected and signed off

9
Old Testing Process

• Advantages
• Thorough Inspection of Changes
• A human eyeballs all differences
• Disadvantages
• Expensive
• Due to tedium, testing is only done when required
• Not Scalable
• Testing time increases significantly as platform
  grows more complex
• Limited Coverage
• No load testing
• Fixed issues are not re-tested
• Errors
• Quality of QA suffers as most testing is manual
  and tedious

10
New Testing ProcessThe metatest Testing
Architecture

• Implemented Circa 2004
• Drivers
• Increasing Complexity
• Need for Automation
• metatest Testing Architecture
• Perl command line engine
• Open source components
• Controls test engines that execute parameterized
  test scripts
• Supports multiple test engines, test
  parallelization (in addition to threading within
  tests), test retargeting and test load variation
• Test scripts execute sequence of steps, with
  assertions made at end of each step.
• Entire test runtime and test script database in
  SCM system (CVS)
• Effort 1 man-months (excluding test scripts)
  with one developer (me)
• Enhanced over a year

11
New Testing ProcessSample Test Script JMeter
Test Engine
Load testing part of same test script
Documents retrieved from TN database to test
document transform assertions

• JMeter 4 test scripts are executed by JMeter
  instances controlled by metatest

12
New Testing Process

• Advantages
• Cheap
• Open-source software, no new hardware cost
• Completely test automation - no tedium
• Scalable
• No time increase with platform complexity - tests
  run in parallel
• Coverage
• Includes load testing
• Can run selective tests, or run regression test
  an entire system in 5 minutes
• No Human Errors
• Quality of QA improves
• New Benefits
• Detailed test results, Test Driven Development,
  Continual testing
• Disadvantages
• Typically, only specific assertions are
  automatically proven
• All differences not examined this may be
  mitigated by using full-document assertions
  regular expressions adjusting for variances

13
New Testing ProcessNew Benefit Detailed Test
Results

• Automatic archival of detailed results
• Data useful for
• Load and stress performance validation (eg image
  on the right) 5
• Capacity planning
• Trend Detection
• Analysing software component failure rates
• New analytical methods can be applied to stored
  data in future

14
New Testing ProcessNew Benefit Test Driven
Development

• While not Unit Testing, our approach has several
  of the benefits of classical TDD
• Tests and code concurrently developed
• Bugs replicated as tests and fixed
• Bugfix tests added to test database
• Simple tests
• Brief run time for tests
• Immediate testing feedback

15
New Testing ProcessNew Benefit Continual
Testing

• Continual Testing
• Dev/Test/UAT tested nightly
• Test UAT also load tested (Load imposed 4 x
  production load)
• Results emailed to developer list so they see it
  first thing in the morning (see image on right)
• Benefits
• All platforms fully regression and load tested
  each night
• Differential testing identical tests run on
  different systems each night. When the results
  differ, bugs are identified by comparing the two
  systems. In example at right, failures are only
  on test in load testing
• The load imposed in load testing is updated to
  reflect growth. Thus, bottlenecks are identified
  before they hit production.

16
Implementation Tips

• Documentation very important
• Documentation crucial to getting developer buy-in
• metatest architecture has 35 pages of user
  documentation (in addition to documentation for
  the tests themselves)
• When using open source components, expect bugs
• Interact with developers of the component
• Contribute if possible
• Document problems, link to Bugzilla DBs etc
• Version control everything to do with testing
• Test scripts
• Even test engines binaries
• Advantages Branching/ Merging, new developer can
  be setup in minutes
• Leverage existing infrastructure
• Most software can simulate endpoints. For eg WM
  C1 OnRamp can create a CommerceOne envelope from
  XML
• Get a domain expert to write your tests

17
Future Steps and Trends

• Plans to improve the metatest architecture
• Implementing other types of testing
• Integration with WmDeployer
• Integration with CECM Verification Tool 6
• Other Testing Resources in the WM space
• Customwares WmUnit 1
• Wmusers.com article on testing using JUnit 2
• Solstice Software 3

18
Future Steps and Trends

• webMethods Testing Support
• WM response to group request for testing support
  Currently no plans exist add unit testing or
  test service generation services to the
  developer.
• However, WMs IDE is moving to the Eclipse
  platform. This should open up interesting
  possibilities with Eclipse plug-ins.
• Server support Ideally, on throwing an
  particular exception, a WM server would save the
  pipeline in a testcase format for bug replication
  on development server. (WM tool does this
  already?)

19
References

• 1 Customware website http//www.customware.net/
  
• 2 wmUsers.com testing article
  http//www.wmusers.com/ezine/2003feb_parvanitis_1
  .shtml
• 3 Solstice software website http//www.solstice
  software.com/
• 4 JMeter website http//jakarta.apache.org/jmet
  er
• 5 Article on analysis of JMeter logfiles in
  stress testing
• http//wiki.apache.org/jakarta-jmeter/LogAnalysis
  
• 6 CECM change verification tool described in
• http//www.wmusers.com/software/uploads/2820B2B_C
  hange_Management_Presentation.ppt