An Automation Experience

1
An Automation Experience October 28,
2004 Presented by Chris Brown
2
Who Am I?

• Chris Brown
• (Chris.Brown_at_cymbal.com)
• QA Manager for Cymbal Corporation
• Responsible for QA Automation Services Delivery
  at Cymbal
• Over 10 Years of Software Testing Experience in
  various industry verticals - Telecommunications,
  Aerospace, e-Business

3
Who is Cymbal?
Cymbal is a global services company
competitively providing the highest quality IT
Consulting, Systems Integration, and Outsourcing
solutions to communications service providers and
media businesses worldwide.

• Founded in 1998, HQ in Fremont, CA
• Offices in North America, Europe, Asia
• 100 Referenceable
• Fortune 1000, industry-leading clients
• Global Delivery
• Onsite, Offsite, Offshore and hybrid options for
  expedited delivery
• Highly Technical Team, focus on Telecom Industry
• 600 seasoned professionals from leading Global
  1000 telecom firms
• Competitive offshore development centers

4
Some satisfied customers
5
Case Study Large MVNO

• Mobile Virtual Network Operator
• A cutting-edge pre-paid mobile services company
  that leverages it brand
• Started Software Development in January 2002,
  Launched Service in July 2002
• QA Started with a 5 Person Team, has grown to 20
  Person Team Today
• Systems Divided into 4 Channels
• CRM Siebel
• Website
• IVRU
• Handset/WAP
• Complex Architecture Siebel, BEA, Oracle,
  Telcordia, Genesys and integration to multiple
  3rd party systems

6
The Need for Automation

• 100 Manual Testing for the first year
• Regression Suite started at 500 test cases
• Monthly Releases since launch
• Regression Suite grows by 50 test cases each
  Release
• Time Required for Testing of the system grows
• Time Allotted for Testing of the system does not
  grow
• Automation Implemented to complete Regression
  Testing of the system within the allotted time

7
Test Tool Selection

• Segue chosen as Automation Tool Vendor
• Silk Test chosen as the Functional Test Tool

8
Phased Approach to Automation
Generate Missing Test Cases (Standard Template)
Test Case Identification and Prioritization
Test Coverage Matrix
Library
Project
Data
Test Script Authoring
Proof of Concept
Test Plans
Options
Scripts
Validation
Phase II
Phase III
Test Automation Framework
9
Test Automation Architecture
10
Test Case Identification/Prioritization
11
Tracking/Reporting Progress
Regression Matrix
Automation Progress
12
Automation Stats/Details

• 5 Person dedicated QA team focused on Automating
  BVTs, Regression and Load Tests
• Flexible Test Framework created using Segues
  SilkTest
• Hybrid delivery strategy test harness designed
  and built onsite, scripting done offshore, test
  validation and execution done onsite
• Automation coverage 70
• Savings in manual automation times for every
  release 30 (compared with domain experienced
  tester, much higher when compared with new-hires)

Representative
13
Challenges

• Execution
• Who Does It?
• Automation Expertise
• Training
• Support
• Maintenance
• Automation Fears Change
• The System Changes
• Moving Baseline
• Data
• Distributed
• Centralized
• Reporting
• Justifying your existence
• ROI

14
Best Practices

• Have a Plan
• Use a Dedicated Automation Team
• Analyze Design Before Coding
• Conduct Code Reviews Walkthroughs
• Have Coding Standards
• Employ Reuse and Modularity
• Have Change Control
• Emphasize Communication

15
The End

• Thank You