Continuous Testing in an Agile Environment

1
Continuous Testing in an Agile Environment

• Nan Krull
• Manager, Software Quality

2
Who is Macrovision?

• Macrovision enables businesses reliant on the
  deployment of software or content to secure,
  adapt and optimize their offering among digital
  distribution channels and destination devices.

• NASDAQ MVSN, founded 1983, public since 1997
• Global leader in Digital Value Management
  Solutions
• Best-of-breed solutions include technologies such
  as
• FLEXnet (Software Licensing)
• InstallShield/InstallAnywhere (Software
  Distribution)
• ActiveReach / Trymedia / eMeta (DRM)
• 70,000 customers
• 1,200 patents
• Offices in the US, UK, France, Germany, Korea,
  Hong Kong, Japan

3
FLEXnet AdminStudio

• Enterprise suite of applications for repackaging,
  customizing, testing and deploying application
  packages in Windows Installer (MSI) environments
• Enterprise clients include international banks,
  airlines, software companies their needs are
  complex and urgent.
• AdminStudio must maintain the highest quality
  standards in order to exceed client expectations.

4
Approaches to Quality

• Linear, Sequential
• QA planning occurs late
• No test case review
• Emphasis on bug fixing

5
Approaches to Quality Continuous Testing

• QA begins at Requirements, Design
• QA test planning parallels Dev
• Feature Test Plan reviewed
• Feature Test Execution starts sooner
• Cost-effective bug removal
• Requires integrated QA and Dev effort

6
Approaches to Quality
7
Good but not good enough

• Defects fixed after Feature Development is
  complete
• The big picture view of certain risk areas may
  not be clear until the end of Feature Development
• Late defect fixing creates risk of breakage
• Late identification of defects and breakage means
  a higher number of defects in the released product

8
Approaches to Quality Agile Environment

• QA begins at Requirements, Design
• QA test planning parallels Dev
• Feature Test Plan reviewed
• Feature Test Execution starts sooner
• Cost-effective bug removal
• Requires integrated QA and Dev effort
• In an Agile environment, as the process is
  compressed these benefits are accelerated.

9
Agile Environment

• Granular User Stories, not Features
• Daily stand-up meetings keep information flowing
• Code is tested as soon as its done
• Code is fixed before more code is written
• Agile may appear more casual but executing
  Agile requires discipline, communication and
  teamwork.

10
Continuous Testing in an Agile Environment
11
What difference does really it make?

• Significantly improved product quality
• Reduced Risk
• Production Cost Savings

12
Assessing Quality

• 83 of bugs originate in the Requirements and
  Design stages of a project.
• Approx. half of software development effort is
  spent on testing.

Quality Assurance Institute, 2006 Economics of
Continuous Testing, Stephen A. Bender, The
Quality Connection, 2006
13
AdminStudio Improved Product Quality
14
AdminStudio Improved Product Quality
15
Economics of Continuous Testing
Economics of Continuous Testing, Stephen A.
Bender, The Quality Connection, 2006
16
Agile maximizes Continuous models

• Early QA involvement prevents defects
• Fixing defects costs less than releasing them
• Preventing defects cost less than fixing them!
• Agile results in earliest possible ID of
• Critical Issues
• Breakage
• Development gets immediate feedback on quality of
  each piece of code and fixes it before coding
  the next piece!
• System Test results in more stable builds and
  less time handling issues

17
Results

• Higher quality product
• Lower risk of releasing defects
• Lower defect handling costs
• Earlier flagging of issues which impact the
  schedule
• Better Quality Satisfied Customers, Increased
  Demand
• Everybody Wins!