Testing in the Cloud

1
Testing in the Cloud

• Our experience of testing software that is out
  there

2
Introduction

• The Presenters
• Paul Selway paul_selway_at_redpathcg.com
• Scott Rostal scott.rostal_at_thrivent.com
• What is The Cloud?
• What is different / the same for cloud testers?
• Testing Philosophy / Test phases
• What was easier / harder than expected?
• Key success factors
• Questions?

3
What is The Cloud?

• Internet (The Cloud) accessible solution
• Off-premise multi-tenant architecture
• Accessed via a browser (PC or Wireless)
• Delivered as Software as a Service (SaaS)
• Common cloud companies are
• Google, Amazon, WorkDay, and Salesforce

4
What things are the same about testing cloud
solutions?

• Requirements coverage based testing
• Risk-based structured test approach
• Test Plan / Test Phases
• Test Cases / Test Data / Test Automation
• Defect Management / Functional Test
• Use of off-shore resources / Virtualization
• Politics Quality / Schedule / Resources

5
What is different about testing cloud solutions?

• Shared multi-tenant test environments
• Security (Test User IDs / SSO)
• Integration of on/off premise systems
• Performance / volume test
• Defect isolation
• Documentation of Out Of the Box reqs
• Release Management

6
Project Philosophy

• Risk-based structured testing approach std
• Exploratory Testing to discover the out of the
  box cloud solution features non-std
• Data Migration Moving legacy data off premise
  Non-Std
• Performance / volume test only our new
  infrastructure not the cloud app Non-Std

7
Testing Phases
Data Migration
Inspections Reviews
End-To-End Integration
Unit / Code Coverage
Exploratory System
UAT Pilot
Load Performance
Overlapping Test Phases
A mix of waterfall (with heavy phase
overlapping), and iterative delivery
8
What was harder than expected?

• Isolating the architectural components for a
  performance test
• Understanding that a cloud solution vendor
  delivers new releases which you have to take!
• Accepting that you have a black box in your
  environment
• Determining who owns defects
• Data Migration Mock Runs

9
System Integration Architecture
10
What was easier than expected?

• Getting the IT leaders to embrace change
• Overlapping Test phases
• End User Adoption
• Use of off-shored test resources
• Functional
• Performance
• Data ETL

11
Key Success Factors

• Learn the Out of the Box cloud application
  features, security model, and configuration
  capabilities
• Understand the holistic system architecture
• Understand the migrated test data
• Establish tight communications with the vendor

12
Questions?
13
Contact Info

• Scott Rostal scott.rostal_at_thrivent.com
• Performance Lead, Thrivent Financial for
  Lutherans
• Paul Selway Paul_Selway_at_redpathcg.com
• President, Red Path Consulting Group