Software Testing - a necessary evil

1
Software Testing - a necessary evil

• SQNZ April 2002
• Michael Osborne
• mike_at_osborne.gen.nz

2
Does this guy know what hes talking about?

• Testing is about bugs
• This guy knows about bugs
• In 25 years of development hes created
  thousands of them
• Now, I pay other people to do this

3
(No Transcript)
4
Simply

• Quidquid latine dictum sit, altum sonatur.
• Whatever is said in Latin sounds profound.
• Sedulously eschew obfuscatory hyperverbosity and
  prolixity.
• Lets all keep it as simple as possible.
• If you dont understand - interrupt!

From - How To Write Unmaintainable Code
http//mindprod.com/unmain.html
5
Whats going on here?

• Why this presentation?
• enquire vs. inform
• tomorrow vs. yesterday
• quality vs quality
• continuous improvement
• how to get and give good testing bang for buck

From - Joel on Software http//www.joelonsoftware.
com
6
What we will cover

• The Principles
• Quality
• Software Quality
• Quality Testing
• The Necessity
• The Evil

• Techniques
• Types of testing
• Testing 2002
• If time permits
• well do some testing

7
Why its necessary

• Analysts developers are imperfect
• Capers Jones - bug input is fps to power of 1.2
• there is no perfect prevention - yet
• you cant ship (successfully) software that
  doesnt work
• its a process measure
• in a highly sophisticated environment
• it goes from primarily being a QC tool
• to a primarily being process measure

8
Why its evil

• You cant test quality in
• You cant test it all
• You cant prove it works
• Its costly and inefficient
• The point is to prove it doesnt work
• The more problems found the better

9
Testing Explained

• Testing is about finding bugs

10
Which part dont you understand?

• If youre not going to find any bugs why are you
  doing it?
• A test that reveals a problem is a success. A
  test that did not reveal a problem was a waste of
  time.
• Testing Computer Software by Kaner, Falk, Nguyen

11
Its an Economics Problem

• Cost of Quality Cost of Conformance Cost
  of Non-Conformance
• Testing adds to Cost of Conformance
• It must directly reduce Cost of Non-Conformance
  or else it is waste
• Testing to do anything other than find bugs is a
  WOMBAT
• a Waste of Money Brains and Time

12
How it works - roughly
13
When CoNC is high...

• Its appropriate to raise the CoC

14
High quality software

• It is possible to develop very high quality
  software
• An example
• NASA space shuttle on-board flight software
• Approx. 3 million lines of code
• lt1 error per 10k LOC after release
• Cost the order of 1000 / LOC

15
The Q word

• quality vs. quality
• its important were clear about this
• it will show the necessity for Software Testing
• it will expose some evils of Software Testing
• So, what is quality?
• And, what is quality?

16
What is software development?

• Putting bugs into software
• ie. hiding bugs in software
• Finding the bugs
• Taking them out again
• Putting (hiding) new / different bugs in when you
  take others out
• Finding those bugs
• Taking them out again
• And so on ...

17
What is better software development?

• Putting fewer bugs into software
• exposing bugs in software
• Finding the important bugs faster
• Taking important bugs out again
• assessing the associated risk
• Putting fewer new / different bugs in when you
  take others out
• And so on
• Todays focus is on Software Testing

18
What is better software testing?

• What is quality software testing?
• Quality is
• conformance to requirements
• Note test cases are not test requirements
• Finding the important bugs faster?
• which leads to the critical question -

19
When is testing done?

• Typically -
• when the testing schedule ends
• when all the test cases have been exercised
• when all the bugs have been found and fixed
• If you dont have a set of testing requirements
• How do you answer the question?

20
The Zero Defects Approach

• Quality conformance to requirements
• Test exhaustively
• Find all defects
• Fix all defects
• Result - quality software
• a quick digression into Zero Defects
• a personal view

21
Direction vs. Destination

• Current state - some level of defects
• Desired state
• more defects
• same level of defects
• fewer defects
• An asymptote
• Note six sigma is not zero
• Zero defects is a pointer and an accelerator
• use it that way

22
Uh oh - theres more than one requirement!!

• Quality conformance to requirements
• There are generally three sets of requirements
• Functionality
• Schedule
• Quality
• The bad news
• they compete

23
The Good-Enough Software Triangle

• Conformance to all requirements

Schedule Delivery Date
Quality Absence of Defects
Functionality Features
24
Software Testing - A Wicked Problem

• A mess is a complex problem with a simple
  solution
• A wicked problem is a simple problem with a
  complex solution
• usually because of stakeholder differences
• Each project (testing problem) potentially has a
  different solution

25
So?

• How do we do it?

26
Software Testing 2002

• Context-driven testing
• all at www.satisfice.com
• Exploratory testing
• heuristics
• short bursts
• focus on results
• purposeful wandering
• you must think

27
Testing Types

• white box - glass box, structural
• path testing - coverage
• black box
• top-down, bottom-up
• unit
• function
• system
• integration
• regression

• performance
• alpha
• beta
• compatability
• quicky
• port
• usability
• exploratory
• hallway
• others ...

28
Bug removal strategy
Important
Find Fix Immediately
Ignore
Urgent
29
Usability Testing

• Graphical User Interfaces coupled with complex
  processes create usability issues
• whats clear and obvious to designer/programmer
  is meaningless to users
• Doesnt require a lot of people
• 5 to 7 is adequate
• Observe, and listen
• understand how they see things
• why they do what they do
• Is your interface clear and consistent?

30
Why usability is critical

• It doesnt matter if it processes everything
  right
• if they dont like it (cant/wont use it, it
  failed)
• Prototype, prototype, prototype
• Consistent vs. clever
• at all times
• Users will be forgiving with a highly usable
  system
• ask Microsoft

31
Hallway Testing

• A simple form of usability testing
• Open up development to interact more with users
• Approach a person (in your organisation) at
  random
• invite for 5-10 minute testing session
• see what they do with your software from no
  background
• is it understandable, usable

32
The evil necessity

• Its all about finding bugs
• A good test is a test that finds a bug
• Be clear about what represents quality
• situation by situation
• Establish the context
• and be guided by it
• Good software testing is a challenging
  intellectual process.
• Go for it!