Testing

1
Testing

• Testing Process
• Testing Activities
• Testing Organization

2
Testing Process

• Establishment of a Test Plan for the project
• Review and approval of the Test Plan
• Start execution of the Test Plan
• Monitor the execution status against the plan
• Report on the execution status against plan
• Take actions to correct the course or the plan,
  if necessary.
• Report on the changes or corrections and gain
  necessary approvals.
• Post test analysis.

3
Test Plan

• Clear and measurable entry criteria for each test
  phase
• Clear and measurable exit criteria for each test
  phase
• Problem Fix Management
• Problem type definition
• Problem severity definition
• Fix cycle by type and severity
• Fix integration and retest criteria
• Test scenario
• Definition
• Estimate of number
• Test cases
• Definition
• Estimate of number
• Estimate of coverage
• Test Schedule(s)
• Test Resource
• Skills and number of people and job assignments
• Test tools (hardware and software)
• Test Management

4
Test Phase Entry-Exit criteria

• For Each Test Phase (Functional, Component,
  System) define the type of testing is to be
  performed
• For Each Test Phase define the entry criteria
• Functional code/help completed and locked in
  the system unit test problems all resolved all
  test cases reviewedetc.
• Component the functional modules all locked in
  the system all functional test problems
  resolved all documentation and data integrated
  no functional stubs within the component needed
  all test scenarios and test cases reviewed etc.
• System all components are locked no
  component stubs within the system hdw and sftw
  system environments set all test scenarios and
  test cases reviewed etc.
• Exit Criteria Defined for Each Test Phase
• All test problems resolved
• All fixes integrated and all the materials are
  locked
• Test problem and resolution data reported to
  management

5
Problem Type Definition

• Functional
• Missing function
• Not working to requirements or design
• Wrong data results
• Erroneous or missing data validation
• User Interface
• Missing or erroneous field size
• Incomplete or erroneous choices
• Incorrect position or color
• No help text or field help
• Incorrect heading or titles
• Erroneous or incomplete messages
• System
• Operating system interface
• Network environment interface
• Database interface
• Documentation and Information
• Incomplete or erroneous statement
• Erroneous example

6
Problem Severity Definition

• Determine the number of severity levels
• Define the severity levels
• Severity 1 system crashes or there is a major
  function working incorrectly and there is no
  workaround or the data in the database is
  corrupted and the system can not be used
• Severity 2 there is a major function working
  incorrectly but there is a work around or the
  query or report results are not correctly
  formatted or sized but the data is accurate or
  the message or documentation is incorrect
• Severity 3 there is a minor functional problem
  and there is a work around the user interface
  has wrong terminology or confusing message
• Severity 4 minor cosmetic looks problem or
  functionally inconvenient but not incorrect

7
Problem Fix Cycle

• Problem Report form filled with appropriate
  problem description and severity defined
• Problem report submitted to development and
  logged
• Development assessed the problem and logs the
  estimated fix schedule.
• Development returns the fix and completes the
  problem report form with fix date, person, etc.
• Problem fixes are integrated into the test system
  and retested and logged as fixed or not.
• If fixed then the fix is placed in a fix bucket
  to be integrated into the real locked system
  later.
• If not the problem reporting-fix cycle repeats
  itself
• Note the implication of a code library and
  test problem/configuration management system

8
Testing Activities

• Test Scenario Development
• Map out the major functions/components/systems
• Relating back to the requirements
• Relating back to the design, if appropriate
• Describe each scenario and the related data for
  that scenario
• Describe what and how each scenario is to be
  tested
• Test Case Development
• Description of each test case
• Input data
• Functional state with assumed precondition and
  needed stubs
• Expected output
• Test case design and generation
• Boundary value testing
• Equivalence class testing
• Logical Relationship testing with decision tables
• Structural and path testing
• Gorilla testing

9
Testing Activities

• Test Execution
• Test environment set up (hardware, software
  data stubs)
• Actual running of the test cases
• Manual
• Automated tool
• Combination of both
• Record the test results
• Analyze the problems
• Report and submit the problems for fixes
• Receive and integrate the fixes, possibly for
  retest

10
Test Data Gathering, Reporting Analysis

• Define the data that need to be collected and
  analyzed
• Severity
• Type
• Arrival
• Backlog
• Design the data gathering mechanism/scheme
• Get concurrence on the data gathering process
• Gather test data by phases Functional,
  Component, System
• Organize and analyze the data
• Within this project
• Across past history and like projects
• Evaluate against entry and exit criteria for each
  phase
• Report the data and the result of the analysis
• Make recommendations based on the analysis
• Can you release the product?

11
Test Schedule

• Developing a schedule
• Task breakdown
• List all tasks, including preparation tasks
• Task units assignable to people
• Task units assignable to units of time (hourly,
  daily, etc)
• Arrange by pre-req and co-req sequences
• Milestones identified and included
• Reviewing and gaining schedule agreement
• Include all participating organizations
• Include upper management and end users

12
Test Resource

• Test resources must map to the test activities
• Test resources must have the skills
• Systems (Win, UNIX, Apple, etc.)
• DB (relational ,etc.)
• Tools (tracking, conf mgmt, coverage, auto test,
  program development, etc.)
• Personal (verbal/written) communications
• Test Processes
• Application and the industry in which the
  applications fit in

13
Test Management

• Appoint a QA/Test Manager
• Develop the Test Process
• Develop the Test Plans
• Review process/plans and gain concurrence with
  the rest of the management team
• Recruit and hire the needed skills and resources
• Monitor the execution against the plan, goal,
  entry and exit criteria.
• Assess risks and goals as testing is in progress
• Take actions as necessary
• Reevaluate and modify Test Process/Test Plans as
  necessary
• Help in Making the Release Decision

14
Test Case Design

• Boundary Value Testing
• Works well with independent input variables
• For each variable there are 6 combinations
• Min, min-1, min1
• Max, max-1, max1
• So for each test input case, there are 6 more
  additional boundary test cases.
• e.g. 15 variables will require (15 6 90)
  boundary cases
• Boolean variable is a special case really no
  boundary situation
• Robustness testing
• Starting with Max value and see how far beyond
  Max can we go
• Starting with Min value and see how far beyond
  Min can we go

15
Test Case Design

• Equivalence Class Testing
• Based on completeness of coverage but not
  duplicative
• Partition the test cases into equivalence classes
  to match the functions
• Classes are mutually disjoint
• Union of classes is the entire set that covers
  all the functions
• e.g. partitioning some date function over the
  months of year (not including boundary)
• One of even month, less February
• One of odd month
• One of leap year
• One of regular February month

16
Test Case Design

• Logical Combination and Decision table
• A technique to list the logical conditions and
  combinations of logical conditions for each of
  the test cases.
• e.g.
• condition 1 T T F F
• condition 2 T F T F
• test case 1 X X
• test case 2 X
• test case 3 X
• Number of logical combinations can be very big
  for n conditions there are 2n combinations.

17
Test Case Design

• Structural and Path Testing
• This falls in the white box testing because we
  are looking inside the design/code
• For each code segment or functional area
• Count the number of predicates, P.
• The total number of paths P 1.
• This will give us the number of test cases to
  cover all the paths.

P 2 of paths 21
3
1
2
18
Test Case Design

• Gorilla Testing
• Utilize the special knowledge of the various
  people with no planned test cases
• Short term (days or weeks)
• Include industry knowledgeable consultants
• Customer support personnel
• Trainers and sales personnel
• Still must organize the proper reporting and
  recording of problems found
• Hard to retest the fixes because the test
  resources may not be able to come back to retest

19
Testing Organization

• Where does the test organization fit in the
  over-all organization ?
• Should be an independent group separate from the
  development group
• Should report to the same level person equal to
  that of the development group
• Must be properly funded
• 1 tester for every 3-4 developers
• Test tools