Title: Jeff Carver
1Improving Software Inspections by Using Reading
Techniques
-
- Jeff Carver
- University of Maryland
- Forrest Shull, Victor Basili
- Fraunhofer Center for Experimental Software
Engineering, Maryland -
2Inspection outline
Software Artifact
Reading Techniques
1
PlanningForm
Planning
organizer
Defect Report Form
2
Detection
inspector
3
Defect Collection Form
moderator inspectors author
Collection
4
Defect Correction Form
Corrected Software Artifact
Software Inspection
Correction
author
3USC Assignment
- Outline of the assignment
- Take short quiz/questionnaire today.
- Perform inspections of the material as you have
for other assignments. Data collection forms are
(mostly) the same. - Use the PBR approach during your individual
preparation, for system requirements (SSRD
sections 3.2 and 5). - Make sure to use the appropriate process!
(tester/user) - Record time spent on PBR separately.
- Jot down the issues you come across during
preparation so you can remember them at the team
meeting. - Perform the team meeting as you always have.
- Feedback interview during ARB.
- Grading is based on
- conformance to the process you were assigned
- NOT number of issues/defects found!
4Software Reading techniques
Reading Technique for Inspections
Question How does one prepare for
inspection? Answer 1 By reading the
document. 2 By understanding what the
document describes. 3 By checking the
required quality properties.
Problem
We often do not know how to read the document!
5Reading techniques
- Developers learn how to write requirements,
design, code, etc. But, they do not learn how to
read them. - Little technical support for the reading process
is currently available.
Reason
Solution
- Provide well-defined and tailored reading
techniques.
Benefit
- Increase the cost-effectiveness of inspections
- Provide models for writing documents of higher
quality - Reduce human influence on inspection results
(i.e., ensure a more engineering approach in
inspections)
6Reading Technique for Inspections
Software Reading Techniques
A reading technique is a series of steps taken to
become acquainted with a document. This helps a
reviewer to detect defects in a software product
throughout the preparation phase of an inspection.
What is it?
- Reading can be done at different levels of rigor
- Ad-hoc
- Checklists
- Scenario-Based Reading (e.g. PBR)
7PBR Concepts Operational scenarios
- What is it? Perspective-Based Reading (PBR) is a
set of reading techniques focusing on points of
view. - The idea is that a document is correct if any
potential stakeholder does not find a defect in
it. Thus, a document should be read from various
point of views (perspectives) - Motivation
- make sure important perspectives are identified
- make each reviewer responsible for particular
perspectives - allow reviewers to build up expertise in
different aspects of the document - History
- Piloted at NASA/Goddard Space Flight Center
- Ongoing studies with students/professionals
through universities UMCP, UMBC, Uni-Kl, NTU,
Uni-Lund, - Adapted for industrial use/training at Allianz
Life Insurance, Bosch Telecom, Robert Bosch GmbH
8PBR Concepts Perspectives
- An example of perspectives Each user of a
requirements document has his or her own needs
for it - a designer, who uses it to produce a system
design - a tester, who produces a test plan to ensure that
the system meets the requirements - a user, who has to make sure the requirements
adequately capture the functionality needed in
the final system
9PBR ConceptsTeam Review
- PBR assigns different perspectives to different
inspectors, i.e., each inspector of an inspection
team reads a document from a particular
perspective. - Benefits
- focus the responsibilities of each inspector
- minimize overlap among inspector responsibilities
- maximize union of defects found
PBR coverage
Ad hoc coverage
10PBR Concepts Operational Scenarios
- The ideas behind PBR are implemented in an
operational scenario for each perspective. - a procedural description of activities and
questions that guides inspectors through the
inspected documented - represented as a set of steps to be followed
- activities consist of
- constructing a model Create an abstraction of
the document under review in order to increase
understanding. - answering questions about the model Focused on
defect classes. - Results
- issue lists
- a high-level model of the system
11PBR Concepts
- Summary so far PBR is built on a number of
concepts (perspectives, team review, procedural
guidelines, defect taxonomy) - Each of these concepts can be altered to conform
to a particular organization - able to add new perspectives of interest to the
organization - able to change the mix of perspectives
- if I expect a predominance of defects of a
certain type, add more reviewers using relevant
perspectives - able to change the level of detail incorporated
into the scenario - able to change the underlying defect taxonomy
12USC Assignment
- Assignment of roles
- Moderator (prepare as normally)
- Author (user) think about how functionality
arranged, level of detail - Reader (user) PBR procedure helps fulfill team
role - paraphrase in terms of functionality for the user
- Tester1 (tester) PBR procedure helps fulfill
team role - should read through all requirements
- responsible for verifying testability of half the
requirements (chosen by moderator) - Tester2 (tester) same as Tester1
13Software defects
- In a generic sense, defects arise when the
development work being done doesnt match the
software specifications already developed or
would cause problems downstream.
14Requirements Defects
- Interpretation of defect
- defect any quality property that is not
fulfilled - avoid focusing on correctness as the one and only
quality property
15Requirements defects
- Example requirements Gas Station Control System
(GSCS) - Overview
- ... The gas station allows customers to purchase
gas (self-service) or to pay for maintenance work
done on their cars. Some local businesses may
have billing accounts set up so that the business
is sent a monthly bill, rather than paying for
each transaction at the time of purchase. There
will always be a cashier on-duty at the gas
station to accept cash payments or perform system
maintenance, as necessary. - The requirements in this excerpt ...concern how
the system receives payment from the customer. A
customer has the option to be billed
automatically at the time of purchase, or to be
sent a monthly bill and pay at that time.
Customers can always pay via cash or credit card.
16Requirements defects
- Functional Requirement 5
- If payment is to be made by cash, the cashier is
responsible for accepting the customers payment
and making change, if necessary. When payment is
complete, the cashier indicates this on the
cashiers interface. The GSCS and the gas pump
interface then return to the initial state.
What is the purchase price?
- Information was lost during the creation of the
requirements. - To handle a cash transaction, the cashier must
know what the purchase price was. - This information has been left out of the
description of the functionality - therefore we
have a defect!
17Requirements defects
- Functional Requirement 3
- If the customer has selected to pay at the time
of purchase, he or she can choose to pay by cash
or credit card. If the customer selects cash, the
gas pump interface instructs the customer to see
the cashier. If the customer selects credit card,
the gas pump interface instructs the customer to
swipe his or her credit card through the credit
card reader. If an invalid or no selection is
made, the GSCS will use the credit card payment
option, which is the default.
Is this the correct response?
- Information was translated incorrectly
- In the example, domain knowledge should indicate
that defaulting to credit card payment is an
incorrect response. (What kind of transaction
ever happens this way?) - Because we know that this functionality should
not be implemented the way it is described, we
have a defect.
18Requirements defects
- Functional Requirement 2
- After the purchase of gasoline, the gas pump
reports the dollar amount of the purchase to the
GSCS. The maximum value of a purchase is 999.99.
The GSCS then causes the gas pump interface to
query the customer as to payment type. - Functional Requirement 4
- If payment is to be made by credit card, then the
card reader sends the account number to the GSCS.
If the GSCS receives an invalid card number, than
a message is sent to the gas pump interface
asking the customer to swipe the card through the
card reader again. After the account number is
obtained, the account number and purchase price
are sent to the credit card system, and the GSCS
and gas pump interface are reset to their initial
state. The purchase price sent can be up to
10000.
?
- Information was described inconsistently
- Because we dont know from domain knowledge which
of the two descriptions is correct, we have found
a defect.
19Requirements defects
- Functional Requirement 6.1
- The customer must give the billing account number
to the cashier, who then enters it at the
cashiers interface. If a valid billing account
number is entered, then the billing account
number, purchase price, and a brief description
of the type of transaction is logged. If an
invalid billing account number is entered, an
error message is displayed and the cashier is
prompted to enter it again. The cashier must also
have the option to cancel the operation, in which
case the cashiers interface reverts to showing
the purchase price and the cashier can again
either receive cash or indicate that monthly
billing should be used.
- Information could be translated incorrectly
- What information is stored?
- What does transaction type mean?
- Gas / Maintenance?
- Full / Partial?
- Credit card / Cash / Monthly bill?
20Requirements defects
- Functional Requirement 7
- To pay a monthly bill, the customer must send the
payment along with the billing account number.
The cashier enters monthly payments by first
selecting the appropriate option from the
cashiers interface. The GSCS then sends a
message to the cashiers interface prompting the
cashier to enter the billing account number, the
amount remitted, and the type of payment. If any
of these pieces of information are not entered or
are invalid, payment cannot be processed an
error message will be displayed, and the
cashiers interface will be returned to the
previous screen. If the type of payment is credit
card, the credit card account number must also be
entered, and then the paper credit card receipt
will be photocopied and stored with the rest of
the years receipts.
Outside scope of system
- Extraneous Information Introduced
- Information that may itself be correct
information causes a problem because it should
not be part of the system. - How could we test for this?
21The tester scenario
- A tester needs to determine if the system
- does what the customer specified that it should
do - follows any guidelines or restrictions the
customer specified - and all this information should be in the
requirements! - Any testing requires
- selecting what is to be assessed (functionality,
reliability, performance, ) - deciding how to assess it
- developing test cases
- determining the correct result of the test cases
- executing the test cases
- comparing the actual results to the correct
results
22The tester scenario
- One testing strategy black-box (functional)
testing - Tries to answer Does a module perform the
functionality it was intended to perform, as
stated in the requirements? - feed the system with data
- observe the reaction
- observed reaction different from expected gt
defect!
White box (structural)
Black box (functional)
inputs
inputs
module
module
outputs
outputs
23The tester scenario
- Equivalence partitioning
- a type of black-box testing
- Examines the set of all possible inputs to
determine which data items are similar to one
another on some relevant dimension. - Sets of similar data items are called equivalence
sets - Helps manage the number of test cases necessary,
since need at least one test case from each
equivalence set. - (Within an equivalence set, one item is as good
as any other - if output is correct for one,
should be correct for all.) - Often used with boundary value testing
(experience indicates faults are more likely for
items near boundaries of equivalence sets).
24The tester scenario
- Using equivalence partitioning in requirements
reviews - When you cant continue following the procedure
because necessary information has been omitted - it can signify a defect
- it can signify information that gets filled in at
a later development phase - the real significance requires you to make a
judgement call Is the missing information
something that must be filled in at this stage of
system development? - The resulting test cases
- are useful NOW only as a tool to help detect
defects - can be transformed LATER into real test cases for
the system
25The tester scenario
- Equivalence partitioning Example 1
- Module keeps a running total of amount of money
received - Input AmountofContribution, specified as being a
value from 0.01 to 99,999.99 - 3 equivalence classes
- Values from 0.01 to 99,999.99 (valid)
- Values less than 0.01 (invalid)
- Values greater than 99,999.99 (invalid)
- Equivalence partitioning Example 2
- Module prints an appropriate response letter
- Input MemberStatus, specified as having a value
of Regular, Student, Retiree, StudioClub - 2 equivalence classes
- Values of Regular, Student, Retiree, StudioClub
(valid) - All other input (invalid)
26The tester scenario
- Equivalence partitioning Example 3
- Input user enters Size, a whole number between 1
and 48 - Multiple equivalence classes
- Values less than 1 (invalid)
- Values in the range 1 to 48 (valid)
- Values greater than 48 (invalid)
- Whole number values (valid)
- Fractional values (invalid)
- Numeric characters (valid)
- Nonnumeric characters (invalid)
27The tester scenario
- Reviewing using equivalence partition testing
- Step 1
- General instructions For each requirement,
identify the inputs. - Specific instructions (Guidelines for
identifying inputs) - Questions
- Does the requirement make sense, based on your
own domain knowledge and the applications
purpose (OCD sections 2.1, 4.1, 4.2, 4.3, 4.4)? - Do you have all the information necessary to
identify the inputs to the requirement? Based on
your domain knowledge and the system capabilities
(OCD section 4.3), are these inputs correct? - Have any of the necessary inputs been omitted?
- Are any inputs specified which are not needed for
this requirement? - Is this requirement in the appropriate section of
the document?
28The tester scenario
- Step 2
- General instructions For each input, construct
equivalence sets. - Specific instructions (Guidelines for
constructing the equivalence sets for an input
class) - Questions
- Do you have enough information to construct the
equivalence sets for each input? Can you specify
the boundaries of the sets at an appropriate
level of detail? - According to the information in the requirements,
are the sets constructed so that no value appears
in more than one set? - Do the requirements state that a particular value
should appear in more than one equivalence set
(that is, do they specify more than one type of
response for the same value)? Do the
requirements specify that a value should appear
in the wrong equivalence set?
29The tester scenario
- Step 3
- General instructions For each equivalence set,
create test cases and specify the expected
output. - Specific instructions (Guidelines for
constructing specific test cases based on system
requirements) - Questions
- Do you have enough information to create test
cases for each equivalence set? - Are there other interpretations of this
requirement that the implementor might make based
upon the description given? Will this affect the
tests you generate? - Is there another requirement for which you would
generate a similar test case but would get a
contradictory result? - Can you be sure that the tests generated will
yield the correct values in the correct units?
Is the resulting behavior specified appropriately?
30A tester scenario example
- Functional Requirement 2
- After the purchase of gasoline, the gas pump
reports the dollar amount of the purchase to the
GSCS. The maximum value of a purchase is 999.99.
The GSCS then causes the gas pump interface to
query the customer as to payment type.
31Form A - Equivalence Sets and Test Cases for New
Requirements Reviewer Name Document Reviewed
2 dollar amount of purchase
Requirement Number Page Description of
Input Valid Equiv. sets Test
cases Test result Invalid Equiv.
sets Test cases Test
result Description of Input Valid Equiv.
sets Test cases Test result Invalid E
quiv. sets Test cases Test result
valid dollar amts. negative dol
lar amts. dollar amts. gt 999.99
100, 13.95 -5.00 1000.00
instruct gas pump interface to query for
payment type display error display
error at cashier int. at cashier int.
32A tester scenario example
- Functional Requirement 7
- To pay a monthly bill, the customer must send the
payment along with the billing account number.
The cashier enters monthly payments by first
selecting the appropriate option from the
cashiers interface. The GSCS then sends a
message to the cashiers interface prompting the
cashier to enter the billing account number, the
amount remitted, and the type of payment...
33Form A - Equivalence Sets and Test Cases for New
Requirements Reviewer Name Document Reviewed
7 cashier input
Requirement Number Page Description of
Input Valid Equiv. sets Test
cases Test result Invalid Equiv.
sets Test cases Test
result Description of Input Valid Equiv.
sets Test cases Test result Invalid E
quiv. sets Test cases Test result
monthly payment option invalid
options
(cant determine) (cant determine)
34A tester scenario example
- If you
- cannot answer a question
- find an inappropriate answer
- consider whether to record an issue in your log
for discussion at the team meeting.
35The user scenario
- User needs to make sure requirements adequately
capture the functionality needed in the system - One strategy for handling functionality is use
cases - Use cases represent the users view of the system
(not an implementation view). - A use case describes a particular set of system
functionality by modeling the dialogue a user
undertakes with the system. - A use case is based on a descriptive scenario of
how the user interacts with the system. It
identifies events that can occur and describes
the system response. - A use case is a complete and meaningful flow of
events, taken from the perspective of a
particular user of the system. - Taken together, all use cases constitute all
possible ways of using the system.
36The user scenario
User customer Start Finish selecting services
System computes the fee and displays the amount
on the control panel
Customer makes the payment
System resets
System indicates that customer should wait wash
is currently busy
System indicates that customer should drive into
car wash
37The user scenario
- Use Cases May Help in Finding Requirements
Defects - Use cases provide a different representation of
the same functionality described in
English-language requirements. - Assumptions for defect detection
- If the English-language requirements are
well-specified, we should be able to convert them
into use cases relatively easily. - If there are hidden problems with the
English-language requirements, they may prevent
us from creating a clear, complete use case
translation. These problems would probably also
cause difficulties in constructing other
artifacts based on the requirements, such as a
design or code. - The process of constructing use cases can be used
as an aid in identifying defects.
38The user scenario
- First major component participants
- entities in the environment (i.e. external to the
system) - that interact with the system to achieve some
functionality. - (often known as actors).
- Participants can be human users, other systems,
external organizations, external devices, etc.,
which interact with the system. - Some participants initiate events others
interact with the system as a result of other
events. - When users are involved a participant is a role
(i.e. a characteristic way of interacting with
the system) NOT a specific person.
39The user scenario
- Second major component Functionality
- More than anything else, use cases are concerned
with describing system behavior. - Problem Very hard to specify an exact definition
for the intuitive way in which we usually
describe system functionality. - Lot of variation in how formally use case
concepts are applied to describing a system very
loosely in some situations, very rigorously in
others. - We ask for functionality to be described at 2
levels - product functions (What features does the product
have? What are the specific activities it can
do?) - use cases/user scenarios (What can customers use
the system for?)
40The user scenario
- Step 1
- General instructions Identify the participants
in the requirements. - Specific instructions (guidelines for
recognizing participants) - Questions
- Q1.1 Are multiple terms used to describe the same
participant in the requirements? - Q1.2 Is the description of how the system
interacts with a participant inconsistent with
the description of the participant (OCD 3.5,
4.5.2, 4.7.1)? Are the requirements unclear or
inconsistent about this interaction? - Q1.3 Have necessary participants been omitted
according to your domain knowledge or the system
description (OCD 3.5, 4.5.2, 4.7.1)? That is,
does the system need to interact with another
system, a piece of hardware, or a type of user
that is not described? - Q1.4 Is an external system or a class of users
described in the requirements which does not
actually interact with the system?
41The user scenario
- Which user groups use the system to perform a
task? - Which user groups are needed by the system in
order to perform its functions? - Which are the external systems that use the
system in order to perform a task? - Which are the external systems that are managed
or otherwise used by the system in order to
perform a task? - Which are the external systems or user groups
that send information to the system? - Which are the external systems or user groups
that receive information from the system?
42Form A - Participants and use cases in new
requirements Reviewer Name Document
Reviewed
Participants
System Functionalities
Involved in which use cases?
1.
F1.
Credit card reader Credit card system Gas
pump Gas pump interface Cashier
interface Customer Cashier
2.
F2.
3.
F3.
4.
F4.
5.
F5.
6.
F6.
7.
F7.
8.
F8.
9.
F9.
10.
F10.
11.
F11.
12.
F12.
13.
F13.
14.
F14.
43The user scenario
- Step 2.
- General goal Describe system functionality.
- Specific goal (Procedure for identifying
specific system product functions and
understanding how they contribute to use cases)
44Form A - Participants and use cases in new
requirements Reviewer Name Document
Reviewed
Participants
System Functionalities
Involved in which use cases?
1.
F1.
Update inventory Bill at purchase Bill on
monthly bill Accept credit card payment Accept
cash payment Receive monthly bill
Credit card reader Credit card system Gas
pump Gas pump interface Cashier
interface Customer Cashier
2.
F2.
3.
F3.
4.
F4.
5.
F5.
6.
F6.
7.
F7.
8.
F8.
9.
F9.
10.
F10.
11.
F11.
12.
F12.
13.
F13.
14.
F14.
45Form B - Use Case Reviewer Name Document
Reviewed
Use Case Name Pay for gas
Use Case Number 1
User Customer Start Replace nozzle
Involves which functionalities? 2 3
4 5
System displays price
System asks customer to select payment
46The user scenario
- Step 2.
- Questions
- Q2.1 Are the start conditions for each use case
specified at an appropriate level of detail? - Q2.2 Are the class(es) of users who use the
functionality described, and are these classes
correct and appropriate? - Q2.3 Is there any system functionality that
should be included in a use case but is described
in insufficient detail or omitted from the
requirements? - Q2.4 Has the system been described sufficiently
so that you understand what activities are
required for the user to achieve the goal of a
use case? Does this combination of activities
make sense, based on your domain knowledge and
the system capabilities (OCD 4.3)? Does the
description allow more than one interpretation as
to how the system achieves this goal? - Q2.5 Do the requirements omit use cases that you
feel are necessary, according to your domain
knowledge or the system goals and capabilities
(OCD 4.2, 4.3)?
47The user scenario
- Step 3.
- General goal Cross-index participants and
functionality. - Specific goal (Procedure for helping identify
which participants are involved in which system
functionalities) - Questions
- Q3.1 Is it clear from the requirements which
participants are involved in which use cases? - Q3.2 Based on your knowledge of the domain and
the description of system participants (OCD 3.5,
4.5.2, 4.7.1), has each participant been
connected to all of the relevant use cases? - Q3.3 Are participants involved in use cases that
are incompatible with the participants
description? - Q3.4 Have necessary participants been omitted
(e.g., are there use cases which require
information that cannot be obtained from any
source described in the requirements)?
48Form A - Participants and use cases in new
requirements Reviewer Name Document
Reviewed
Participants
System Functionalities
Involved in which use cases?
1.
F1.
Update inventory Bill at purchase Bill on
monthly bill Accept credit card payment Accept
cash payment Receive monthly bill
1 1 1 1 1 1
Credit card reader Credit card system Gas
pump Gas pump interface Cashier
interface Customer Cashier
2.
F2.
3.
F3.
4.
F4.
5.
F5.
6.
F6.
7.
F7.
8.
F8.
9.
F9.
10.
F10.
11.
F11.
12.
F12.
13.
F13.
14.
F14.
49Contact Information
- Email me with any questions
- carver_at_cs.umd.edu
- and remember to cc the class account
50Bibliography
- Forrest Shull, Ioana Rus, and Victor R. Basili,
How Perspective-Based Reading Can Improve
Requirements Inspections, IEEE Computer, July
2000. - Oliver Laitenberger and Jean-Marc DeBaud, An
Encompassing Life-Cycle Centric Survey of
Software Inspection, Journal of Systems and
Software, 2000. - Oliver Laitenberger, Khaled El Emam, Thomas
Harbich, An Internally Replicated
Quasi-Experimental Comparison of Checklist and
Perspective-based Reading of Code Documents, IEEE
Transactions on Software Engineering, 2000. - Lionel C. Briand, Khaled El Emam, Bernd Freimut,
Oliver Laitenberger, A Comprehensive Evaluation
of Capture-Recapture Models for Estimating
Software Defect Content, IEEE Transactions on
Software Engineering, 2000. - Victor R. Basili, Forrest Shull, Fillipo
Lanubile, Building Knowledge through Families of
Experiments, IEEE Transactions on Software
Engineering, vol. 25, no. 4, pp. 456-473, July
1999. - Carolyn B. Seaman and Victor R. Basili,
Communication and Organization An Empirical
Study of Discussion in Inspection Meetings, IEEE
Transactions on Software Engineering, vol. 24,
pp. 559--572, June 1998. - Forrest Shull, Developing techniques for using
software documents A series of empirical
studies, PhD thesis, University of Maryland,
1998. - Lionel Briand, Khaled El-Emam, Thomass
Fussbroich, and Oliver Laitenberger, Using
Simulation to Build Inspection Efficiency
Benchmarks for Development Projects, in
Proceedings of the Twentieth International
Conference on Software Engineering, pp. 340-349,
IEEE Computer Society Press, 1998. - Victor R. Basili, Evolving and Packaging Reading
Technologies, Journal of Systems and Software,
vol. 38, July 1997. - Victor .R. Basili, Scott Green, Oliver
Laitenberger, Fillipo Lanubile, Forrest Shull,
Sivert Sorumgard, and Marvin V. Zelkowitz, The
Empirical Investigation of Perspective-Based
Reading, Journal of Empirical Software
Engineering, vol. 2, no. 1, pp. 133-164, 1996. - Adam A. Porter, Lawrence G. Votta, and Victor R.
Basili, Comparing Detection Methods for Software
Requirements Inspections A Replicated
Experiment, IEEE Transactions on Software
Engineering, vol. 21, pp. 563-575, June 1995. - A bibliography of more inspection-related
literature can be found at http//www.iese.fhg.de/
Inspections