testing, hosting and managing business applications

1
testing, hosting and managing business
applications
Managing Quality in your ERP Project 12 Mistakes
to Avoid Best Practices to Adopt
Dan Downing, VP Testing Services MENTORA
GROUP Atlanta Boston DC www.mentora.com
August 8, 2006
2
Objectives

• Shine the spotlight on key Quality mistakes that
  ERP implementations should avoid
• Things you will learn
• The 12 Mistakes to Avoid
• Their tell-tale signs and risks
• Strategies for mitigating their impact
• Tools to enable repeatability

3
ERP Definition

• Strict
• Enterprise Resource Planning
• Financials, Manufacturing, HR, Supply Chain, CRM
• e.g., Oracle 11i, SAP, PeopleSoft
• Looser
• Any business-critical packaged application that
  on which you run a substantial part of your
  business
• e.g., Integrated healthcare, student
  administration, hospital management, customer
  service, brokerage trading, staff recruitment
  placement

4
ERP Challenges

• High-intrusion
• Expensive to implement

• Unique workflows, interfaces
• Often coupled with BPR

• Dont control quality
• Conflicting maintenance cycles

• Complex middleware
• Challenging to tune

5
Competing Stakeholders Key Goals
Budget
ERP Consultant
ERP Vendor
Personal goals
Hosting Provider
Management
Competitive Advantage
Business
QA
IT
Time Money
Cost-savings
6
12 Quality Risks
Test cases
Sponsorship
Functional correctness
Who does testing?
System performance
How much time for testing?
Testing infrastructure
Measuring quality
Automated vs. manual
What tools?
Enough testing?
Test data
7
Mistake 1
Confusing product quality with implementation
quality

• Tell-tale signs
• Industry analysts still bearish on ERP software
  quality
• Software is configured to your workflows,
  interfaced to your surrounding systems, with your
  converted data
• Risk
• Configuration decisions trigger software
  conflicts that yield incorrect results
• Best Practices to Mitigate
• End-end workflow functional testing
• Upgrades Run same tests on both releases
• Resources
• Testing team, test case inventory, automated
  functional testing tools

8
Mistake 2
Trusting your hosting providers environment
assertions

• Tell-tale signs
• Hosting provider hosts no/few similar
  environments (hw, patch level, disk subsystems)
• System configuration based on standard vendor
  recommendations vs. hard statistics
• Risk
• Under-configured environment
• Poor peak load performance
• Best Practices to Mitigate
• Performance test peak transaction volumes
• Resources
• Cross-functional performance team, load testing
  tool, performance expert

9
Mistake 3
Failure to secure strong sponsorship for testing

• Tell-tale signs
• Quality not a corporate priority
• No quality representation on leadership team
• No Quality owner
• No separate testing budget
• Risk
• Rocky go-live at best, major failure at worst
• Best Practices to Mitigate
• Make Quality an integral KPI for each major
  activity of the implementation
• Resources
• Enroll VP of Development or Director of QA to
  develop strategy

6s
10
Mistake 4
Confusing conference room pilot with test case
development

• Tell-tale signs
• The only test cases generated by conference room
  pilot team
• Test cases incomplete coverage, input data,
  expected results
• Risk
• Uneven quality due to inadequate test coverage
  testing the process, not the load
• Best Practices to Mitigate
• Make complete test cases a deliverable by each
  function group
• Resources
• QA, business users, Test Management tool or Excel

11
Mistake 5
Believing that testing can be done by the
implementation team

• Tell-tale signs
• ERP consultants focus is on the software
  supporting the business, not on the
  infrastructure supporting the load
• No separate testing plan
• Business configuration team under-resourced,
  over-stretched
• Risk
• Uncertain functional quality, poor performance,
  no quality metrics for decision-making
• Best Practices to Mitigate
• Plan and staff Testing as a separate subproject
• Resources
• QA team, business users, testing consultant

12
Mistake 6
Failure to allocate enough time for testing

• Tell-tale signs
• Project plan shows testing not starting until
  after configuration and development are complete
• Testing resourced by same people as other
  activities
• Risk
• Uncertain quality due to compressed testing time,
  incomplete coverage, no quality metrics
• Best Practices to Mitigate
• Start test planning, tester training, testing
  setup early in the project
• Resources
• QA team, business users, testing consultant

13
Mistake 7
Failure to make quality visible early-on

• Tell-tale signs
• Quality not a regular leadership topic
• No/weak quality metrics for each phase
• No compelling visuals of quality KPIs
• Risk
• No basis for management decisions based on
  quality uninformed go-live decision
• Best Practices to Mitigate
• Identify key quality metrics by phase graphs to
  summarize supporting detail (early risk areas,
  functional gaps, enhancements later test
  coverage, testing progress, defects)
• Resources
• Test Management tool, Excel

14
Mistake 8
Failure to use the right tools to support testing

• Tell-tale signs
• Test plan assumes testing is manual
• No budget for test tools, testing consultant
• Few defined test cycles
• Testing function but not load
• Risk
• Low test repeatability error-prone fewer test
  cycles incomplete test coverage
• Best Practices to Mitigate
• Strategy that combines automated and manual
  testing budget to support it
• Resources
• Functional, load test management tools expert
  testing consultant to accelerate
• Pre-defined scripts from value-add vendors

15
Mistake 9
Believing that automated testing replaces manual
testing

• Tell-tale signs
• Test plan assumes testing all automated
• Test plan compresses test time on expectation
  that automated is faster
• Risk
• Ineffective and incomplete testing test
  automation black eye based on failed
  expectations uncertain quality
• Best Practices to Mitigate
• Plan that blends selective automated testing,
  whose coverage grows over time
• Resources
• Functional testing tools expert testing
  consultant to implement transfer knowledge

16
Mistake 10
Failure to secure enough testing infrastructure

• Tell-tale signs
• Infrastructure plan has no/limited dedicated
  testing environments
• Insufficient database instances, disk storage
• Volatile, non-production test data
• Risk
• Frustrated test team, low testing productivity
  incomplete testing uncertain quality
• Best Practices to Mitigate
• Plan for at least these environments pilot, user
  acceptance, QA-1 (current release), QA-2 (new
  release), training (2)
• Resources
• QA leadership, infrastructure team, ERP/DB vendor

17
Mistake 11
Underestimation of effort to create reusable
test data

• Tell-tale signs
• No plan for populating / copying / refreshing
  test databases
• Test cases not planned with cascading data
  dependencies understood
• Risk
• Frustration low testing productivity failure to
  meet timeline
• Best Practices to Mitigate
• Plan for disk-to-disk test DB backups sequence
  test cases with macro workflows id data
  dependencies
• Resources
• QA leadership, data expert, Test Management tool

Disk-to-disk DB refresh
Procure to Pay
18
Mistake 12
Believing that youre done with testing once you
go live

• Tell-tale signs
• No operational maintenance plan
• No/low consideration for test reusability
• Not a focus for any one stakeholder
• Not todays concern
• Risk
• Inability to keep up with patches, upgrades risk
  of introducing problems without proper testing
• Best Practices to Mitigate
• Growing, reusable automated test inventory
• Resources
• QA leadership, functional and load testing tools

Then what?
19
Summary

• ERP quality is different
• Scope, complexity, stakeholders, cost, competing
  goals
• Quality is embedded in many activities
• Dont trust a stakeholders quality assertions
• Make it a leadership topic
• Define quality KPIs for each phase
• Graph them!
• Use tools that will enhance testing productivity
• Planning, environments and disk are key
• Automation is not a silver bullet
• Repeatability yields ROI youll leverage this
  after go-live
• No sure way to test system performance without
  tools
• Watch for the tell-tales signs of these common
  mistakes
• Leverage best practices, enroll experts to
  mitigate!

20
Questions?

• ddowning_at_mentora.com
• www.mentora.com