Software Testing

1
Software Testing

• Testing across the entire software development
  lifecyle

2
Software test engineer

• Software test engineer is a professional who is
  in charge of one or more technical test
  activities, including
• designing test inputs
• producing test case values
• running test scripts
• analyzing results
• reporting results to developers and managers

By Ammann and Offutt, 2008
3
Software test manager

• Test manager is in charge of one or more test
  engineers. They
• set test policies and processes
• monitors the processes
• interact with other managers on the project
• help the engineers do their assignments

By Ammann and Offutt, 2008
4
Activities of a software test engineer

• Design tests by creating test requirements
• requirements are transformed into actual values
  and scripts ready for execution
• executable test are run against the software
• the results are evaluated to determine if the
  tests find a fault
• Powerful tool formal coverage criterion
• Provides test engineers ways to decide what test
  inputs to use during testing such that maximum
  possible faults can be found
• Also provides stopping rules

By Ammann and Offutt, 2008
5
The mind of a tester

• Four different kinds of thinking shown by a good
  tester (by Kaner et. al)
• Technical thinking ability to model technology
  and understand causes and effects
• Creative thinking ability to generate ideas and
  see possibilities
• Critical thinking ability to evaluate ideas and
  make inferences
• Practical thinking ability to put ideas into
  practice
• An example of these kinds of thinking is found in
  a fable called The Kings Challenge

By Everett and McLeod Jr., 2007 and Kaner et. al,
2002
6
The mind of a tester (contd)

• The Kings Challenge (a fable)
• Once upon a time, a mighty king wanted to
  determine which of his three court wizards was
  the most powerful
• So he put the three court wizards in the castle
  dungeon and declared whoever escaped from his
  respective dungeon cell first was the most
  powerful wizard in all the kingdom

By Everett and McLeod Jr., 2007 and Kaner et. al,
2002
7
The mind of a tester (contd)

• Here is what each court wizards has done
• The first wizard immediately started chanting
  mystical poems to open his cell door
• The second wizard immediately started casting
  small polished stones and bits of bone on the
  floor to learn how he might open his cell door
• The third wizard sat down across from his cell
  door and thought about the situation for a
  minute. Then, he got up, walked over to the cell
  door and pulled on the door handle. The cell door
  swung open because it was closed but not locked
• Thus, the third wizard escaped his cell first and
  became known as the most powerful wizard in all
  the kingdom
• What kinds of tester thinking did the third
  wizard exercise in solving the kings puzzle?

By Everett and McLeod Jr., 2007 and Kaner et. al,
2002
8
The mind of a tester (contd)

• Answer
• Creative thinking ability to generate ideas and
  see possibilities
• Practical thinking ability to put ideas into
  practice

By Everett and McLeod Jr., 2007 and Kaner et. al,
2002
9
Non-software testing at the user level

• Buying a car
• Use the automobile industry to find non-computer
  testing examples that can be related to software
  testing
• Motivation for testing a car is to determine its
  quality or its functionality before buying one
• As a user, you are not interested in performing
  all possible kinds of test since you assume that
  manufacturer has done those tests for you
• Important to realize that you limit your testing
  some way
• Limited test is referred as test drive
• To better understand the testing limits, examine
  what you do NOT test followed by what you TEST

By Everett and McLeod Jr., 2007
10
Non-software testing at the user level

• Objectives of a Test Drive are NOT
• to break the car
• Seek guarantees and warranties that car
  manufacturer has already proven that car is
  unbreakable under normal driving conditions for
  x thousand miles or y years (reliability)
• to improve the cars design
• If you want design changes in the car, simply
  shop for different model or different
  manufacturer

By Everett and McLeod Jr., 2007
11
Non-software testing at the user level (contd)

• What you test is determined by your
  transportation needs. The needs become test drive
  objectives
• Test objectives are the measurable milestones in
  testing which indicate that the testing
  activities have achieved the desired goals
• Test drive objectives are translated into testing
  approaches that validate whether the car on the
  dealers lot meets your transportation objectives

By Everett and McLeod Jr., 2007
12
Non-software testing at the user level (contd)

• Objectives of a Test Drive ARE
• to validate affordability
• to validate attractiveness
• to validate comfort
• to validate usefulness
• to validate performance
• Each of these objectives can be validated without
  trying to break the car or redesign it.
• Each of these objectives are personal and you
  need to prioritize them to in final your final
  decision

By Everett and McLeod Jr., 2007
13
Non-software testing at the user level (contd)

• Testing approaches include
• examining the sticker price and sale contract
• trying out radio, the air conditioner, and the
  lights
• trying acceleration, stopping, and cornering
• These testing approaches are referred to by
  common terminology in the testing industry
• examine static testing
• (observe, read, review without driving the
  car)
• try out functional and structural testing
• (work different features of the car without
  driving the car)
• try performance testing
• (work different features of the car by
  actually driving the car)

By Everett and McLeod Jr., 2007
14
Non-software testing at the developer level

• Building a car
• Testing objectives of a new car to be built
• validate design via scale models
• validate operation of prototypes
• validate mass assembly plans from prototypes
• Starts with written requirements such as
• Seats six
• Carries five suite cases
• Runs on regular gas
• Consumes gas at a rate of 35 miles per gallon in
  highway
• Has a top speed of 100 miles per hour

By Everett and McLeod Jr., 2007
15
Non-software testing at the developer level
(contd)

• These requirements are the nonnegotiable design
  and manufacturing boundaries set by groups other
  than the designers
• The manufacturer is responsible to build a new
  car which satisfies all the requirements
• New requirements, ? more understandable test
  objectives
• The auto design tester is responsible to validate
  the current state of the new car against its
  requirements
• If the new car doesnt initially meet the
  requirements, then its the designer, not the
  tester who must improve the design for full
  compliance
• After the design changes are done, the tester
  needs to revalidate the revised design against
  the requirements
• Design, test, correct, retest cycle continues
  until the new car meets its requirements and is
  completed before the car is manufactured
• Requirements are essential for testing validation
  at every stage of developing a new car

By Everett and McLeod Jr., 2007
16
Non-software testing at the developer level
(contd)

• Testing approaches used while building new cars
• plan the tests based on requirements and design
  specifications
• examine blueprints and clay models
• perform and analyze wind tunnel tests
• perform and analyze safety tests
• perform and validate prototype features
• drive prototype and validate operations
• Specifications (blue prints or models) are the
  designers interpretation of the requirements on
  how the design can be manufactured
• When specifications are validated against the
  requirements, all the subsequent physical car
  assembly validation can be performed against the
  specifications

By Everett and McLeod Jr., 2007
17
Non-software testing at the developer level
(contd)

• Similar to the test drive, the car builder
  testing approaches can be described by common
  testing terminology
• examine static testing
• (observe, read, review without driving the
  car)
• perform functional and structural testing
• (work different features of the car models,
  mock-ups, and manufactured car subassemblies)
• drive performance testing
• (work different features of the car in the
  prototypes)

By Everett and McLeod Jr., 2007
18
Primary objectives of testing

• Testing objective 1 Identify the magnitude and
  sources of development risk reducible by testing
• Company prepares a business case including
  expected benefits, costs, and risks
• If the project is determined to be bad on
  investment, then the project does not start
• If the benefits outweigh the cost and its good
  return on investment, the benefits are compared
  to the risks
• If the risk is high but the likelihood of the
  risk occurrence is low, the project is on
• If the risk and the likelihood of the risk
  occurrence is high, the following questions are
  asked
• Can this risk be reduced by testing?
• If the risk can be reduced, how much can testing
  reduce it?
• If the risk factors are well known and
  quantifiable, its possible that testing can
  reduce the probability of the risk occurrence

By Everett and McLeod Jr., 2007
19
Primary objectives of testing (contd)

• Testing objective 2 Perform testing to reduce
  identified risks
• Test planning includes
• positive testing (things work as required)
• negative testing (things that break)
• Test planning highlights the risk areas such that
  the largest possible percentage of the test
  schedule and effort (both positive and negative
  testing) are allocated to reduce that risk
• Testing does not completely eliminate the risk
  since there are always more scenarios to test
  than the allotted time and resources to complete
  the tests

By Everett and McLeod Jr., 2007
20
Primary objectives of testing (contd)

• Testing objective 3 Know when testing is
  completed
• Since 100 testing is unrealistic, the tester
  must determine when to stop testing
• The determination should start with the positive
  test items in the test plan
• Complete as much of the risk-targeted testing as
  possible relative to cost benefit break-even
  point
• 100,000 business risk, 50,000 testing cost to
  reduce the risk (NO!!!)
• A rule of thumb is 10-20 cost to benefit
  break-even point for testing
• 100,000 business risk, 1000-2000 testing cost
  (YES!!!)
• Tester must complete as many of the negative test
  items in the plan with the remaining the testing
  budget after positive testing and risk testing
  are completed

By Everett and McLeod Jr., 2007
21
Primary objectives of testing (contd)

• Testing objective 4 Manage testing as a standard
  project within the development project
• Often testing is treated as simple skill that
  anyone can do without planning, scheduling or
  resources
• Since business risk represents real dollar loss,
  real dollar testing is required to reduce the
  risk
• Real dollar testing means that personnel with
  testing experience should be become the testing
  team with access to management, resources and
  schedules necessary to plan and complete the test
• Observations
• The testing does not have to be a hit or miss
• Testers are a limited resource

By Everett and McLeod Jr., 2007
22
References

• P. Ammann and Jeff Offutt, Introduction to
  software testing, Cambridge University Press,
  2008
• G. D. Everett and R. McLeod Jr., Software
  Testing, Wiley Inter-Science, 2007
• C. Kaner, J. Bach, B. Pettichort, Lessons learned
  in software testing, Wiley Computer Publishing,
  Addison-Wesley, 2002, pp. 286