Project Management and Testing

1
Project Management and Testing
2
The Unified Process
Phases
Elaboration
Inception
Construction
Transition
Requirements Analysis Design Implemen- ta
tion Test
Activities
Prelim. iterations
3
PROPS
supporting
operative
managing
4
Prestudy Phase

• Tollgate (TG0) decision to start prestudy
• Requirement engineering
• Interviews of customer and/or product management
• Analysis
• Business case analysis
• Customers
• Competitors
• Costs
• Benefits
• Risks
• Results in an assignment for a feasibility study
• TG1

5
Feasibility Phase

• Tollgate (TG1) decision to start feasibility
  study
• Requirement engineering
• User-interface prototyping, specification, use
  cases, etc
• System design
• Anathomy, architecture, Implementation Proposals
• Simulations using tools or role play
• Project planning
• Risk analysis
• Estimation of size, effort, cost and schedule
• Resources, competence, organization, etc
• Life-cycle model (prototyping, evolutionary,
  waterfall, etc)
• Results in an assignment for an implementation
  project
• TG2

6
This is how far you will go in SMD136!
7
We need to manage the artifacts!
Resource PlanProject Portfolio
Actors Use Cases Use Case Diagrams StoryboardsReq
uirements Specification

• Milestones!
• Activities
• Deliverables
• Resources
• Responsibles
• Dates

Black BoxRequirements
FunctionalSpecification
Diagrams - Class- Sequence- Statechart-
Activity Design Patterns System Design
SystemTest
SystemDesign
Test PlanTest Cases
SystemDevelopment
SystemValidation
Notes and Details Signatures Design Patterns
8
Resource Plan
Resource Plan
Phases and Milestones
Activities and Deliverables
Budget and Resources
9
Project Plan
Project Plan
Introduction - Goal
Organization
Phases and Milestones
Activities and Deliverables
Budget and Resources
Releases
Risk
10
Project Portfolio
Project Portfolio
Project Plan
Resource Plan
System Requirements
System Design
System Test Plan
Quality Plan
Configuration Management

11
Builds
12
Relationship Between Use Cases, Iterations and
Builds
Iteration 5
build 5.3
Use case 14
Use case 23
13
Relationship between Use Cases, Iterations and
Builds
Iteration 5
4
6
5.4
5.2
build 5.3
Use case 7

Use case 14


Use case 23
Use case 9
extends or includes
14
Final Code Build and Integration Schedule a
Banking Example
week 23
week 31
biweekly
daily
weekly code builds
baseline
release
15
Final Code Build and Integration Schedule a
Banking Example
week 23
week 31
biweekly
daily
weekly code builds
baseline
Integration of module into baseline
bank query module
release
tasks
bank deposit module
bank withdrawal module
16
Plan evolutionary increments of the baseline!
week 23
week 31
biweekly
daily
weekly code builds
baseline
Module 1
release
Integration of module into baseline
activitytask
Module 2
Module 3
17
Resource Planning
Month 1
Month 2
Month 3
Month 4
Month 5
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
Milestones
Complete testing
Freeze requirements
Release to production
2
2
2
3
2
2
3
Requirements
Hal vacation
4
4
4
3
3
4
4
4
4
4
4
4
Design
Karen vacation
2
2
2
1
1
1
4
To be assigned
Risk Analysis
4
4
4
4
4
3
3
4
4
4
4
3
3
4
4
4
4
4
4
4
Given team size
18
Cyclic, Incremental or Evolutionary Planning
1
1
1
1
1
1
Week
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
Delivery
Cycle 1
Milestones
Cycle 2
Cycle 3
1. strategy 2. plan 3. requirements 4. design 5.
implementation 6. test 7. postmortem
Iteration 1
Iteration 2
1. strategy
1. strategy
Iteration 3
19
Range of Errors in Estimating Eventual Cost
Range of cost estimates after conceptualization
phase,based on actual cost of 1
4
Conceptualization phase
Range of cost estimates after requirements
analysis phase
1
25c
Requirements analysis
1
Design
1
Implementation
1
Integration/Test
1
20
The Variables of Project Management

• 1. The total cost of the project,
• e.g., increase expenditures
• 2. The capabilities of the product,
• e.g., subtract from a list of features
• 3. The quality of the product,
• e.g., increase the mean time between failure
• 4. The date on which the job is completed.
• e.g., reduce the schedule by 20
• e.g., postpone project's completion date one month

21
The Window of Opportunity
Quality
Functionality
Schedule
Cost
On time, within budget and meet requirements!
22
Bullseye Figure for Project Variables
cost
Target 70K
Target 100
capability
duration
Target 30 wks
defect density
Target 4 defects/Kloc
23
Bullseye Figure for Project Variables
cost
Target 70K
Target 100
Actual 90K
Actual 100
this project
capability
duration
Target 30 wks
Target 4 defects/Kloc
Actual 20 wks
Actual 1 defect/Kloc
defect density
24
Coding versus Documenting
1/3
2/3
25
Urgent versus Important
26
Triage in Project Management

• Among top items in importance?
• if so, place it in do at once category
• otherwise Ignore without substantially affecting
  project?
• if so, place it in last to do category
• otherwise (Do not spend decision time on this)
• place in middle category

27
More on Project Management

• eXtreme Programming Leadership (smd136)
• SMD155 Project in Software Enginnering
• Or, most of the other 4th year projects!
• Dedicated Project Management courses at IES etc.
  Read them!

28
Assignment 5 Resource Plan

• The resource plan should include information on
  which main project phases and activities that are
  planned (further requirements analysis, software
  architecture design, testing, etc) for the rest
  of the project and the amount of resources that
  are planned for each activity. It should also be
  expressed when these activies are planned to be
  active.
• Each main project activity should have a start
  and stop date (it's ok to say 'month1' or 'day30'
  instead of a real date, it may actually be
  preferable).
• Each main project activity should have a planned
  amount of manhours (1 manyear is 1800 manhours).
• Each main project activity should have a number
  of staff members allocated to the activity, with
  one team or activity leader.
• You will propable find that some staff members
  will need to double in some functions or work
  with more than one activity. This means that you
  need to plan the work so that a staff member will
  not be allocated too much or too less.
• Milestones etc should be visible in the plan.

Deadline 5

• Remember that your investor is willing to invest
  5 manyears in the development of the companys
  first system, so the plan should not exceed that
  ammount of work (note that 5 manmonths are
  already spent on your initial project
  portfolio...). How you plan to use the resources
  is up to you, but should be described in your
  resource plan.
• There are licenses of Microsoft Project, which
  you may use to produce Gantt charts etc if you so
  prefer. You can also use Microsoft PowerPoint or
  any general purpose software to do plans
  graphically.
• Keep It Simple! -)

29
Resource Plan
Resource Plan
Phases and Milestones
Activities and Deliverables
Budget and Resources
30
Assignment 6 Project Portfolio

• The project plan is a document that binds
  together the rest of your project documents into
  a project portfolio!
• These are the documents that should be part of
  your project portfolio
• Project Plan including resource planning
• Also on resources already spent (time)
• Requirements Specification
• Design Document
• The project plan is what you need to complete by
  building on your resource plan.
• The project plan should contain information on
  how you intend to execute the project!

Deadline 6

• Some examples
• Example Project Plan 73-93
• Case Study 123 - 131
• Again, Keep It Simple! But not empty! Ask
  yourselves what you need to go on with the
  project! Use your central nervous system!!! -P
• Paper AND electronically!

31
Project Plan
Project Plan
Introduction - Goal
Organization
Phases and Milestones
Activities and Deliverables
Budget and Resources
Releases
Risk
32
Project Portfolio
Project Portfolio
Project Plan
Resource Plan
System Requirements
System Design
System Test Plan
Quality Plan
Configuration Management

33
Testing...
34
Now

• Testing!
• Unit Testing
• Black Box Testing
• White Box Testing
• Test Coverage
• Integration Testing
• System Testing

35
Development Overview
Requirements
loss of information
Customer
loss
Architecture
loss
loss
Interface specs
Detailed design
OK?
loss
loss
Function code
Module (e.g., package) code
System code
36
Golden Rules of Testing

• Goal of testing maximize the number and severity
  of defects found per dollar spent
• thus test early
• Limits of testing Testing can only determine the
  presence of defects, never their absence
• use proofs of correctness to establish absence

37
Testing
Sys Req
HL Design
LLDesign
Code
Test Plan
38
Artifacts and Roles for Integration Testing
Component engineer
Integration tester
System tester
Test engineer
. . . . . . . . . . . . . . . . . . responsible
for . . . . . . . . . . . . . . . . . . . . . . .
. . . . .
Test evaluation
?
Use-case model

Test procedure
Test plan
Test component
Defect management
Test case
39
Testing The Big Picture
3. System tests
Use-cases System Functions
2. Integration tests
Packages of classes
Module combination
1. Unit tests
Combinations of methods in class
Module
Methods
Function
40
Unified Process
Elaboration
Inception
Construction
Transition
Requirements Analysis Design Implemen- ta
tion Test
Prelim. iterations
41
Black-box Testing
Input determined by...
Result
Actual output compared with required
Black box
requirements
42
White-box Testing
Input determined by...
Result
White box
Confirmation of expected behavior
design elements
43
Black-, Gray-, White-box Testing
Result
Input determined by...
Actual output compared with required output
Black box
requirements
Gray box
As for black- and white box testing
requirements key design elements
White box
Confirmation of expected behavior
design elements
44
Test Input Possibilities
principal
100
100M
Infinitely many illegal values!
20
inflation estimate
25
Infinitely many legal values!
interest rate
1
0
Minimize what is needed to test, or youll never
end testing...
45
Covering Every Statement is Not Sufficient

• Code attempt to implement flowchart
• if ( (ugt1) (v0) ) (1)
• x x/u (2)
• if ( (u2) (xgt3) ) (3)
• x (4)
• u2, v0 and x3
• executes every line (1) - (4)
• gives the correct output x 2.5
• However, line (3) is wrong

Required program
ugt1 and v0
Yes
x x/u
No
u2 or xgt0
Yes
x
No
46
Testing Ranges Elementary Cases

• 1. within range
• 2. at the boundaries
• of the range
• 3. outside the range
• (illegal)

range
47
Perform Method Testing 1/2
1. Verify operation at normal parameter values
(a black box test based on the units
requirements) 2. Verify operation at limit
parameter values (black box) 3. Verify
operation outside parameter values (black box)
4. Ensure that all instructions execute
(statement coverage) 5. Check all paths,
including both sides of all branches (decision
coverage) 6. Check the use of all called
objects 7. Verify the handling of all data
structures 8. Verify the handling of all files
48
Perform Method Testing 2/2
9. Check normal termination of all loops (part
of a correctness proof) 10. Check abnormal
termination of all loops 11. Check normal
termination of all recursions 12. Check abnormal
termination of all recursions 13. Verify the
handling of all error conditions 14. Check timing
and synchronization 15. Verify all hardware
dependencies
49
Types of System Tests 1

• Volume
• Subject product to large amounts of input.
• Usability
• Measure user reaction (e.g., score 1-10).
• Performance
• Measure speed under various circumstances.
• Configuration
• Configure to various hardware / software
• e.g., measure set-up time.
• Compatibility
• -- with other designated applications
• e.g., measure adaptation time.
• Reliability / Availability
• Measure up-time over extended period.

50
Types of System Tests 2

• Security
• Subject to compromise attempts.
• e.g., measure average time to break in.
• Resource usage
• Measure usage of RAM and disk space etc.
• Install-abililty
• Install under various circumstances.
• measure time to install.
• Recoverability
• Force activities that take the application down.
• measure time to recover
• Serviceabililty
• Service application under various situations.
• measure time to service
• Load / Stress
• Subject to extreme data event traffic

51
Key Attributes for Usability Testing

• Accessibility
• How easily can users enter, navigate exit?
• e.g., measure by average time taken to . . .
• Responsiveness
• How quickly does the application allow the user
  to accomplish specified goals?
• e.g., measure by average time taken
• Efficiency
• Degree to which the number of required steps for
  selected functionality is minimal
• minimal deduced in theory
• e.g., measure by minimal time / average time
• Comprehensibility
• How easy is the product to understand and use
  with documentation and help?
• e.g., measure time taken for standard queries

52
Alpha- and Beta- Releases

• In-house and highly trusted users
• Multiplies testing
• Previews customer reaction
• Benefits third-party developers
• Forestalls competition

Alpha

• Selected customers
• Multiplies testing activity
• Gets customer reaction

Beta
53
Roadmap for the Transition Iterations

• Define population
• Plan defect collection
• Identify stopping criteria

1. Plan alpha and beta testing.

• Prepare
• Distribute install
• Carry out (users / customers)
• Gather defect reports
• Observe stopping criteria
• Correct defects

2. Conduct alpha testing.
3. Conduct beta testing.
54
Stopping Criteria

• Completing a particular test methodology
• Complete the procedures of a method or tool.
• Estimated percent coverage for each category
• predetermine percent of each how to calculate
• e.g., 95 statement coverage
• Error detection rate
• predetermine rate with given severity level
• e.g., 2 medium severity defects or less per 100
  hours of operation
• Total number of errors found
• (if possible) computed from a percentage of
  remaining defects
• predetermine percent
• e.g., 95 of estimated existing defects found

55
Stopping Criteria Graphical Representation
Percentage tested or found

Target 95
time
56
Stopping Criteria Graphical Representation
Errors found per 1000 hrs
Error detection rate
time
Target lt 7 per1000 hrs for 4 weeks
57
Typical Day-by-Day Code Integration Process
biweekly
daily
weekly builds
week 25
week 26
week 27
week 28
week 29
week 30
week 31
week 24
week 23
release
58
ANSI/IEEE 829-1983 Software Test Documentation
1. Introduction 2. Test plan items under test,
scope, approach, resources,
schedule, personnel 3. Test design items to be
tested, the approach, the plan in detail 4.
Test cases sets of inputs and events 5. Test
procedures steps for setting up and executing
the test cases
6. Test item transmittal report items under test,
physical location of results, person responsible
for transmitting 7. Test log chronological
record, physical location of test, tester name
8. Test incident report documentation of any
event occurring during testing which requires
further investigation 9. Test summary
report summarizes the above
59
Next

• eXtreme Programming (XP) 4/10
• Guest Lectures
• Ronny Pekkari? 6/10
• Bernt Ericsson! 14/10
• Håkan Alm? 13/10
• Seminars Course Evaluation

60
Thats all folks! Thanks for your attention!
61
The Really Big Picture!
order of testing
Requirements
(11) Acceptance tests
Docs.
(10) Installation tests
Architecture
(9) Usability tests
Docs
Docs
(8) System tests
Detailed design
(7) Regression tests
Docs
Interface specs
(6) Integration tests
Function code
Docs
(5) Interface tests
Code
Code
Module (e.g., package) code
(2),(4) Module tests
Code
Code
(1),(3) Function tests
Iteration or System code
Complete code
62
RoadMap for Unit Testing
Requirements
1. Plan for unit testing
Identify largest trouble spots
Detailed design
Unit test plan
2. Acquire test set(test cases)
Products of prior testing
Test set (test cases)
3. Execute unit test
Test results
Code under test
IEEE, 1986
63
Paths to be Checked
Parameter settings make sense?
N
Y
Set _name to defaultName"
Parameter name too long?
Y
N
Truncate name
Set _name to parameter
64
Paths to be Checked
Parameter settings make sense?
N
Y
Set _name to defaultName"
Parameter name too long?
Y
N
Decision Coverage
Truncate name
Set _name to parameter
65
Example Encounter State-Transition Test Sequence
Preparing
Player dismisses qualities menu
Waiting
Move to adjacent area
66
Unit Testing Summary

• Unit testing ? pieces
• Other testing ? assemblies
• Black box input / output only
• White box verifies processing
• Several ways
• Ensure completeness
• Test planning earlier / better
• helps clarify requirements

67
Plan for Unit Testing

• 1. Decide on the philosophy for unit testing
• individual engineer responsible (common)?
• reviewed by others?
• designed performed by others?
• 2. Decide what / where / how to document
• individuals personal document set (common)?
• how / when to incorporate into other types of
  testing?
• incorporate in formal documents?
• use tools / test utilities?
• 3. Determine extent of unit testing (i.e., in
  advance).
• do not just test until time expires
• prioritize, so that important tests definitely
  performed
• 4. Decide how and where to get the test input
• 5. Estimate the resources required
• use historical data if available
• 6. Arrange to track time, defect count, type
  source

68
Performing Class Unit Tests

• 1. Exercise methods in combination
• 2-5, usually
• choose most common sequences first
• include sequences likely to cause defects
• requires hand-computing the resulting attribute
  values
• 2. Focus unit tests on each attribute
• initialize, then execute method sequences that
  affect it
• 3. Verify that each class invariant is
  unchanged (invariant always true, e.g.
  NumberOfPlayers lt5)
• verify that the invariant is true with initial
  values
• execute a sequence (e.g., the same as in 1.)
• verify that the invariant still true
• 4. Verify that objects transition among expected
  states
• plan the state / transition event sequence
• set up the object in the initial state by setting
  variables
• provide first event check that transition
  occurred . etc.

69
Plan and Execute Integration Tests

• 1. Decide how and where to store, reuse and code
  the integration tests.
• show this in the project schedule
• 2. Execute as many unit tests (again) as time
  allows
• this time in the context of the build
• no drivers or stubs required this time
• prioritize by those most likely to uncover
  defects
• 3. Exercise regression tests
• to ensure existing capability has not been
  compromised
• 4. Ensure build requirement are properly
  specified
• 5. Exercise use cases that the build should
  implement
• test against the SRS
• 6. Execute the system tests supported by this
  build

70
Summary

• Integration process executed in carefully planned
  builds
• Integration testing each build
• System testing whole application
• Regression testing verify that changes do not
  compromise pre-existing capabilities

71
5 minutes...
5 MIN