SE 754NT

1
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana Lecture
27 For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu
2
Standard testing approaches

• Software Testing by Ron Patton

3
Infamous software failures

• Disneys Lion King
• 1st multimedia CD-ROM
• Just in time for Christmas sales expectations
  very huge
• CS phones began to ring
• Failure to properly test SW on many different PC
  models
• Worked well on the models that Disney programmers
  used to create the game !!

Software Testing by Ron Patton
4
Intel bug

• Try this on your PC
• (4195835 / 3145727) 3145727 4195835
• If answer is 0, your computer is fine
• Anything else gt you have the Pentium bug !!

Software Testing by Ron Patton
5
NASA Mars Polar Lander 1999
Software Testing by Ron Patton

• Lander disappeared Dec 3rd 1999
• Failure to set a single data bit
• NASA tried to save by replacing radar with
  contact switch
• During testing vibration caused switch to trip
  (like it had landed)
• Lander tested by multiple teams
• Leg fold down procedure (never looked to see if
  touch down bit set)
• Landing process (always reset computer, clearing
  bit, before starting testing

6
What goes into a SW product?
Software Testing by Ron Patton
7
What parts make up a SW product?
Software Testing by Ron Patton
8
Optimal test effort
Software Testing by Ron Patton
9
Precision vs accuracy

• Testing a simulation game like a flight
  simulator..
• Should you test precision or accuracy?
• How about a calculator?
• Depends on what the product is and what the
  development team is aiming at !!

10
Neither accurate nor precise
Software Testing by Ron Patton
11
Precise, but not accurate
Software Testing by Ron Patton
12
Accurate, but not precise
Software Testing by Ron Patton
13
Accurate and precise
Software Testing by Ron Patton
14
Try this

• Start the Windows Calculator program
• Type 5,000-5 (comma is important)
• Look at result
• Is it a bug or not??

15
Quiz

• Q Given that its impossible to test a program
  completely, what information do you think should
  be considered when declaring whether its time to
  stop testing?
• A There is no correct answer when to stop
  testing. Each project is different. Things to
  consider would be, Are lots of bugs being found?,
  Is the team satisfied with the number and types
  of tests that have been run?, Has the product
  been validated against the users requirements?

16
Quiz

• Q If you were testing a flight simulator, what
  would be more important, accuracy or precision?
• A Simulator should look and feel like flying a
  real plane. Whats more important is how
  accurately the simulator reflects reality,
  precision can follow. This is exactly what has
  happened with simulation games over the years.

17
Quiz

• Q Is it possible to have a high-quality and
  low reliability product? What might an example
  be?
• A Yes, but it depends on the customers
  expectations for quality. Example is a high
  performance sports car that has high quality
  (fast acceleration, style, etc) but are
  notoriously unreliable (often breaking down and
  expensive to repair)

18
Quiz

• Q Why is it impossible to test a program
  completely?
• A There are too many inputs, too many outputs,
  and too many path combinations to fully test.
  Also, software specs can be subjective and be
  interpreted in different ways (bug is in the eye
  of the beholder!!)

19
Quiz

• Q If you were testing a feature of your
  software on Monday and finding a new bug every
  hour, at what rate would you expect to find bugs
  on Tuesday?
• A Two axioms The number of bugs remaining is
  proportional to the number of bugs already found
  (wont come in Tuesday and find the SW perfect!).
  Pesticide paradox continuing to run the same
  tests over and over again means you wont find
  more bugs until you start adding more tests!!
  Result youll continue to find bugs at the same
  rate or slightly less

20
Part 2
21
Test to pass and test to fail

• Test-to-pass assume SW minimally works
• Dont push capabilities
• Dont try to break it
• Simple and straightforward test cases
• Would you drive a brand new car model at top
  speed right off the line?
• Run these first

22
Test to pass vs Test to Fail
Software Testing by Ron Patton
23
Equivalence partitioning
24
Example

• How would you test the Save as function?
• Valid characters except \ / ? lt gt and
• 1 to 255 characters

25
Example

• Equivalence partitions
• Valid characters
• Invalid characters
• Valid length names
• Names that are too short
• Names that are too long

26
Data testing
27
Data testing

• Simple view of software is to divide into two
  domains
• Data (or its domain)
• Program
• Examples of data
• Keyboard input
• Mouse clicks
• Disk files
• printouts

Divide testing up along the same lines!!
28
Data testing

• The amount of data can be overwhelming even for a
  simple program
• Must intelligently reduce test cases by
  equivalence partitioning
• Boundary conditions
• Sub-boundary conditions
• Nulls
• Bad data

29
Boundary conditions
30
Simple program with bug
1 Rem Create a 10 element integer array 2 Rem
Initialize each element to 1 3 Dim data(10) As
Integer 4 Dim i As Integer 5 For i 1 To 10 6
data(i) -1 7 Next i 8 End
Software Testing by Ron Patton
31
Simple program with bug
data(0) 0 data(1) -1 data(2) -1 data(3)
-1 data(4) -1 data(5) -1
data(6) -1 data(7) -1 data(8) -1 data(9)
-1 data(10) -1
Software Testing by Ron Patton
32
Types of boundary conditions

• Numeric
• Character
• Position
• Quantity
• Speed
• Location
• Size

33
Characteristics of those types

• First/last
• Start/finish
• Empty/full
• Slowest/fastest
• Largest/smallest
• Next to/farthest from
• Min/max

• Over/under
• Shortest/longest
• Soonest/latest
• Highest/lowest

34
Testing the boundary conditions

• First-1/Last1
• Start-1/Finish1
• Less then empty/More than full
• Even slower/Even faster
• Largest1/Smallest-1

• Min-1/Max1
• Just over/Just under
• Even shorter/Longer
• Even sooner/later
• Highest1/Lowest-1

35
Example

• Test entry field that allows 1 to 255
• Try entering 1 character
• Enter 255 characters
• Try 254
• Enter 0
• Enter 256

36
Program that reads and writes a floppy

• Try saving a file with one entry
• Try saving a file at the limits of what a floppy
  disk holds
• Try saving an empty file
• Try saving a file that is too big for the floppy

37
Program to print multiple pages onto a single
page

• Try printing one page (the standard case)
• Try printing the most pages allowed
• Try printing zero pages
• Try printing more than it allows

38
Flight simulator

• Try flying at ground level
• Try flying at maximum height
• Try flying below ground level
• Below sea level
• Outer space

39
Sub-boundary conditions

• Some boundary conditions are internal to the
  software and not user apparent
• Powers of two
• Bit (0-1), nibble (0-15), byte (0-255), word,
  kilo, mega, giga, etc
• ASCII table
• Not nice and contiguous

40
Default, empty, blank, null, zero, and none

• Rather than type wrong information for example,
  no data is entered (just press return)
• Forgotten in spec and overlooked by programmer
• Happens in real life!
• SW should default to lowest value or some
  reasonable value

41
Default, empty, blank, null, zero, and none

• Always have an equivalence partition for these
  values

42
Invalid, wrong, incorrect, and garbage data

• Test-to-fail
• See if the SW can handle whatever a user can do
• Reasonable to think that some percentage of
  people will handle SW incorrectly
• If any data lost, for whatever reason, user will
  blame SW (bug!)

43
Invalid, wrong, incorrect, and garbage data

• If SW wants numbers, give it letters
• If SW expects positive numbers, give it negative
  numbers
• Fat fingers
• How many windows can you have open
• No real rules, be creative, have fun!

44
State testing

• A software state is a condition or mode that the
  SW is in
• Other side of testing looks at programs logic
  flow
• States
• Transitions between them
• Same type of complexity required partitioning
  this space as well

45
Create a state transition map
Software Testing by Ron Patton
46
State diagram parts

• Each unique state that the SW can be in
• Input or condition that takes it from one state
  to the next
• Set conditions and produced output when a state
  is entered or exited

47
Reducing the number of states

• Covering all paths is a traveling salesman
  problem. Instead
• Visit each state once
• Test common state-to-state transitions
• Test least common state transitions
• Test error states and returning from error states

48
What to specifically test

• Check all state variables
• Static conditions, values, functionality
  associated with being in that state or moving to
  and from the state
• May be something visible like a window or dialog
  box
• May be invisible such as a part of a
  communication program or financial package

49
Example dirty document flag
50
Testing state to fail

• Race conditions and bad timing
• Saving and loading the same document at the same
  time with two different programs
• Sharing the same printer, com port, or peripheral
• Using different programs to simultaneously access
  a common DB

51
Testing state to fail

• Repetition, stress, and load (not handling the
  worst case scenario)
• Repetition same operation over and over (looking
  for memory leaks)
• Stress running SW under less than ideal
  situations (low memory, low disk space, slow
  CPUs, etc)
• Analyze SW for external resource needs
• Test by limiting these to bare minimum
• Goal is to starve SW (one type of boundary)

52
Testing state to fail

• Load testing opposite of stress testing
• Instead of starving, you force feed (more than it
  can handle)
• Largest possible data files
• Connect as many printers,peripherals as possible
• Time (running over long periods) is another form
  of load testing

53
Issues

• Others may not be receptive to efforts to break
  SW this way
• no customer will use this way
• Test automation is required to make some of this
  practical

54
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana End of
lecture For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu