Writing a Test Strategy

1
Writing a Test Strategy

• Tor Stålhane

2
Why a testing strategy

• We need a testing strategy to help
• System and software testers plus
  Test-and-Evaluation staff to determine the
  overall test strategy for development or
  modification of a software intensive system
• Project stakeholders customers and senior
  management approve the test strategy
• Testers and system and software analysis to
  determine
• Test objectives
• Qualification requirements
• Verification and validation criteria

3
Testing strategy concepts

• We will discuss the following concepts
• Purpose of a test strategy
• Testing focus
• Contents of a test strategy
• Software integrity levels
• Test objectives and priorities

4
Purpose of a test strategy 1

• The test strategy is important to
• Obtain consensus on test goals and objectives
  from stakeholders e.g. management, developers,
  testers, customers and users
• Mange expectations right from the start
• Be sure that we are heading in the right
  direction
• Identify the type of tests to be conducted at all
  test levels

5
Purpose of a test strategy 2

• When we write a test strategy is important to
  remember that
• Whatever we do some kind of test strategy will
  emerge. Thus, we might as well specify the one we
  think is best
• A documented strategy is the most effective way
  to get an early agreement on goals and objectives
• We need to address
• Human factors usability
• Interoperability, except for stand-alone systems

6
Testing focus

• Our focus will depend on which stakeholder we are
  considering at the moment
• Users acceptance test and operational tests
• Analysis systems test, qualification tests
• Designer integration tests
• Programmer unit tests
• The main point is that we need to define the
  stakeholder first then the tests to be run.

7
Contents of a test strategy 1

• The following is a list of what can be specified
  in a test strategy. Not all of it is needed in
  all cases only use what is necessary.
• Project plan, risks and activities
• Relevant regulations depending of application
  area
• Required processes and standards
• Supporting guide lines

8
Contents of a test strategy 2

• Stakeholders e.g. users, testers, maintainers
  and their objectives
• Necessary resources people, computers
• Test levels and phases
• Test environment
• Completion criteria for each phase
• Required documentation and review method for each
  document

9
Software integrity level

• There are several ways to define software
  integrity levels. When we choose an integrity
  level this will strongly influence the way we do
  testing.
• We will look at three definitions of integrity
  levels
• IEEE 1012 general software
• ISO 26262 automotive software
• IEC 61508 general safety critical software

10
IEEE 1012 general software

• 4, High some functions affect critical system
  performance.
• 3, Major some functions affects important
  system performance
• 2, Moderate some functions effect system
  performance but workarounds can be implemented to
  compensate.
• 1, Low some functions have noticeable effect on
  system performance but creates only inconveniences

11
Vv Activities
VV activity Development requirements level Development requirements level Development requirements level Development requirements level Design level Design level Design level Design level Implementation level Implementation level Implementation level Implementation level Test level Test level Test level Test level
SW Integrity Level 4 3 2 1 4 3 2 1 4 3 2 1 4 3 2 1 1
Acceptance test execution X X X
Acceptance test plan X X X
Interface analysis X X X X X X X X X
Management and review support X X X X X X X X
Management review of VV X X X X X X X X X X X
12
ISO 26262 automotive software

• The ASIL level A, B, C or D is the outcome of
  the combination of three factors
• S Severity. How dangerous is a event
• E Probability. How likely is the event
• C Controllability. How easy the event to
  control if it occurs

13
Finding the ASIL level
Severity Probability C1 C2 C3
S1 E1 QM QM QM
S1 E2 QM QM QM
S1 E3 QM QM A
S1 E4 QM A B
S2 E1 QM QM QM
S2 E2 QM QM A
S2 E3 QM A B
S2 E4 A B C
S3 E1 QM QM A
S3 E2 QM A B
S3 E3 A B C
S3 E4 B C D
14
Methods for software integration testing
Methods and Measures Methods and Measures According to req. ASIL ASIL ASIL ASIL
Methods and Measures Methods and Measures According to req. A B C D
1 Requirements based test 9.4.4
2 External interface test 9.4.4
3 Fault injection test 9.4.4
4 Error guessing test 9.4.4
15
Methods for software unit testing
Methods and Measures Methods and Measures According to req. ASIL ASIL ASIL ASIL
Methods and Measures Methods and Measures According to req. A B C D
1 Functional tests 8.4.2 See table 8.2 See table 8.2 See table 8.2 See table 8.2
2 Structural coverage 8.4.2 See table 8.3 See table 8.3 See table 8.3 See table 8.3
3 Resource usage measurement 8.4.2
4 Back-to-back test between simulation model and code, if applicable 8.4.2
16
IEC 61508 safety critical software
PFDavg Ft / Fnp . The table above, together
with this value decides the SIL level.
17
Detailed design
Technique/Measure Technique/Measure Ref SIL1 SIL2 SIL3 SIL4
1a Structured methods including for example, JSD, MASCOT, SADT and Yourdon C.2.1 HR HR HR HR
1b Semi-formal methods Table B.7 R HR HR HR
1c Formal methods including for example, CCS, CSP, HOL, LOTOS, OBJ, temporal logic, VDM and Z C.2.4 --- R R HR
2 Computer-aided design tools B.3.5 R R HR HR
3 Defensive programming C.2.5 --- R HR HR
4 Modular approach Table B.9 HR HR HR HR
5 Design and coding standards Table B.1 R HR HR HR
6 Structured programming C.2.7 HR HR HR HR
7 Use of trusted/verified software modules and components (if available) C.2.10 C.4.5 R HR HR HR
Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied. Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied.
18
Module testing and integration
Technique/Measure Technique/Measure Ref SIL1 SIL2 SIL3 SIL4
1 Probabilistic testing C.5.1 --- R R HR
2 Dynamic analysis and testing B.6.5 Table B.2 R HR HR HR
3 Data recording and analysis C.5.2 HR HR HR HR
4 Functional and black box testing B.5.1 B.5.2 Table B.3 HR HR HR HR
5 Performance testing C.5.20 Table B.6 R R HR HR
6 Interface testing C.5.3 R R HR HR
a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level. a) Software module and integration testing are verification activities (see table A.9). b) A numbered technique/measure shall be selected according to the safety integrity level. c) Appropriate techniques/measures shall be selected according to the safety integrity level.
19
Test objectives and priorities

• Only in rather special cases can we test all
  input binary input output and few parameters.
  Thus, we need to know
• The overall objective of testing
• The objective of every test case
• The test case design techniques needed to achieve
  our goals in a systematic way.
• The test objectives are our requirements
  specification for testing.

20
Test data selection

• One of the important decisions in selecting a
  test strategy is how to select test data. We will
  look at five popular methods
• Random testing
• Domain partition testing
• Risk based testing
• User profile testing
• Bachs heuristic risk-based testing

21
Random testing

• The idea of random testing is simple
• Define all input parameters e.g. integer, real,
  string
• Use a random test / number generator to produce
  inputs to the SUT
• The main problem with this method is lack of an
  oracle to check the results against. Thus, manual
  checking is necessary.
• The method is mostly used for crash testing
  (robustness testing) will the system survive
  this input?

22
Domain partition testing 1

• Definitions
• A domain is a set of input values for which the
  program performs the same computation for every
  number of the set. We want maximal domains so
  that the program performs different computations
  on adjacent domains
• A program is said to have a domain error if the
  program incorrectly performs input classification

23
Domain testing simple example
If a gtlt 0, this equation has the following
solution
Otherwise we have that
24
Testing domains
25
Risk based testing

• The idea of risk based testing is to
• Identify the risk or cost of not delivering a
  certain functionality.
• Use this info to prioritize tests. We will cover
  this in more details later under Test
  prioritation

26
User profile testing

• The main idea with this type of testing is to
  generate tests that mirror the users way of
  using the system.
• Consider a situation where we know that the users
  in 80 of all case
• Fetch a table from the database
• Update one or more info items
• Save the table back to the database
• Then 80 of all tests should test these three
  actions.

27
Bachs risk-based testing

• Bachs heuristics is based on his experience as a
  tester. Based on this experience he has
  identified
• A generic risk list things that are important
  to test
• A risk catalogue things that often go wrong
• We will give a short summary of the first of
  Bachs lists.

28
Bachs generic risk list 1

• Look out for anything that is
• Complex large, intricate or convoluted
• New no past history in this product
• Changed anything that has been tampered with or
  improved
• Upstream dependency a failure here will cascade
  through the system
• Downstream dependency sensitive to failure in
  the rest of the system
• Critical a failure here will cause serious
  damage

29
Bachs generic risk list 2

• Precise must meet the requirements exactly
• Popular will be used a lot
• Strategic of special importance to the users or
  customers
• Third-party developed outside the project
• Distributed spread out over time or space but
  still required to work together
• Buggy known to have a lot of problems
• Recent failure has a recent history of failures.

30
Test and system level 1
31
Test and system level 2

• From the diagram on the previous slide we see
  that we can test on the
• Electronics level e.g. DoorActuator sends the
  right signal
• State / signal level e.g. door is closed iff
  DoorStateClosed
• Logical level e.g. the door remain closed as
  long as the speed is non-zero
• Safety level e.g. the door remain closed as
  long as the train is moving

32
Acknowledgement

• The first part of this presentation is mainly
  taken from Gregory T. Daichs presentation
  Defining a Software Testing Strategy, 30 April
  2002.