An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection

1
An Experimental Comparisonof Checklist-Based
Readingand Perspective-Based Readingfor UML
Design Document Inspection

• Giedre Sabaliauskaite
• Fumikazu Matsukawa
• Shinji Kusumoto
• Katsuro Inoue
• Graduate School of Engineering Science, Osaka
  University
• Graduate School of Information Science and
  Technology, Osaka University

2
Contents

• Background
• Research Goals
• Experiment description
• Reading techniques
• Experimental design
• Data analysis
• Results
• Conclusions and Further Work

3
Software Inspection

• Software inspection is an effective and efficient
  method to detect defects in early stages of
  software development life-cycle
• Software inspection usually consists of several
  activities 1

Planning
Defect detection
Defect collection
Defect correction
1 O. Laitenberger, J.M. DeBaud, An
encompassing life cycle centric survey of
software inspection, The Journal of Systems and
Software, vol. 50, no. 1, 2000, pp. 5-31.
4
Reading techniques

• Ad hoc and Checklist-based Reading are the most
  popular 1
• Ad hoc no guidance for inspectors
• Checklist-based Reading (CBR) Checklist
• Scenario-based Reading is a more recent approach
  Scenario
• Perspective-based Reading (PBR) 2 is one of
  Scenario-based approaches
• Software product is inspected from different
  perspectives (designer, tester, etc.)
• Provides inspectors with different Scenario for
  each perspective
• We analyzed CBR and PBR inspection techniques
  during experiment

1 O. Laitenberger, J.M. DeBaud, An
encompassing life cycle centric survey of
software inspection, The Journal of Systems and
Software, vol. 50, no. 1, 2000, pp. 5-31. 2
V.R. Basili, S. Green, O. Laitenberger, F.
Lanubile, F. Shull, S. Sorumgard, M.V. Zelkowitz,
The Empirical Investigation of Perspective-Based
Reading, Empirical Software Engineering An
International Journal, vol.1 , no. 2, 1996, pp.
133-164.
5
Related research

• Several works are done in the area of Unified
  Modelling Language (UML) diagram inspection
• One of the works is described in 1
• Experimental comparison of CBR and PBR
• 18 subjects (practitioners) and 2 software
  systems
• UML diagrams inspected Class, State, Sequence,
  Collaboration
• Results - PBR 3-person teams exhibited higher
  effectiveness and lower cost per defect than CBR
  teams

• Authors of different studies came to the
  conclusion that UML inspections need to be
  further investigated

1 O. Laitenberger, C. Atkinson, M. Schlich, K.
El Emam. An experimental comparison of reading
techniques for defect detection in UML design
documents. The Journal of Systems and Software,
53 183-204, 2000.
6
Research Goals

• Conduct UML diagram inspection experiment in
  Osaka university
• Language used during experiment - Japanese
• Compare Reading techniques CBR and PBR with
  respect to
• Defect detection effectiveness
• Cost per defect
• Time spent on inspection
• Compare both individual inspector results and
  team results
• Acquire experience for future inspection

7
Structure of CBR and PBR techniques, used during
experiment

• CBR
• Diagram-specialized Checklist developed
• Included 20 questions
• PBR
• Three perspectives User, Designer, Implementer
• Three scenarios (one for each perspective)
  developed
• Scenarios consisted of several steps, each of
  them included
• Explanations
• Tasks to perform
• Questions to answer

8
Assignment of UML diagrams

• All inspectors were given system requirement
  description and Use-Case diagrams
• Defects were injected into Activity, Class,
  Sequence and Component diagrams

Document Seminar system Hospital system CBR PBR perspective PBR perspective PBR perspective
Document Seminar system Hospital system CBR User Designer Implementer
System requirement description 1 1 ? ? ? ?
Use-Case diagrams 1 1 ? ? ? ?
Activity diagrams 8 7 ? ?
Class diagrams 1 1 ? ? ? ?
Sequence diagrams 12 7 ? ? ? ?
Component diagrams 1 1 ? ?
9
Experiment design

• Subjects
• 59 third year students of the Software
  Development course, Osaka University
• Objects
• Seminar information system ( 24 pages )
• Hospital information system ( 18 pages )

PBR PBR PBR CBR
User Designer Implementer CBR
Seminar system 7 6 6 11
Hospital system 7 6 6 10
10
Defects

• Three types of defects inserted into UML diagrams
• Syntactic missing or unnecessary information
• Semantic misrepresentation of a concept, or
  unclear representation
• Consistency inconsistency between UML diagrams
• In total, 15 defects inserted into the each
  software system
• 4 into Activity diagrams
• 3 into Class diagrams
• 5 into Sequence diagrams
• 3 into Component diagrams

11
Experimental Hypotheses

• H01 subjects who use PBR technique should spend
  less time on inspection
• H02 subjects who use PBR should have higher cost
  per defect
• H03 individual defect detection effectiveness
  should be different between CBR and PBR
  inspectors
• H04 team defect detection effectiveness should
  be different between CBR and PBR inspector teams

12
Data analysis

• Two types of analysis
• Individual data analysis
• Defect detection effectiveness
• Time spent on inspection
• Cost per defect
• Team data analysis
• Subjects assigned into 3-person virtual teams
• Effectiveness of CBR and PBR teams compared

13
Percentage of inspectors, who have detected
defects relevant to different perspectives
14
Defect detection effectiveness (according to
defect types)
15
Defect detection effectiveness (according to UML
diagram types)
16
Time spent on inspection, Cost per defect and
Defect detection effectiveness
Variables Reading technique Mean SD
Time spent on inspection CBR 62.9 11.7
Time spent on inspection PBR 51.3 15.1
Cost per defect CBR 6.2 1.6
Cost per defect PBR 10.2 3.5
Effectiveness CBR 70.2 11.5
Effectiveness PBR 69.1 15.3
17
Virtual teams

• Subject assignment into 3-person virtual teams
• Any three CBR reviewers are included into team
• One reviewer using each of the three PBR
  perspectives included into team

18
Comparisons between CBR and PBR teams
CBR
PBR
Comparisons
C1 C2 C3
C4 C5 C6
C7 C8 C9
U1 D1 I1
U2 D2 I2

U6 D6 I6
CBR group 1
PBR group 1
C1 C2 C3
C4 C5 C6
C7 C8 C10
U1 D1 I6
U2 D2 I1

U6 D6 I5
CBR group 2


PBR group 2

19
Team data analysis (1/2)

• CBR and PBR teams were compared with respect to
  Team defect detection effectiveness

• Number or comparisons, in which either CBR or PBR
  was more effective, or both techniques showed
  similar effectiveness

Software system Team defects detection effectiveness Team defects detection effectiveness Team defects detection effectiveness
Software system CBRgtPBR PBRgtCBR CBRPBR
Seminar 54 753 326 688 585 743 712 544 449 600
Hospital 10 160 482 176 8 064 149 760
20
Team data analysis (2/2)

• T-test with significance level of 2.5 was used
  to evaluate in which comparisons there was a
  significant difference between CBR and PBR team
  effectiveness
• Two types of comparisons made
• All defects included
• Syntactic defects omitted

Comparison type Software system Defect detection effectiveness Defect detection effectiveness Defect detection effectiveness
Comparison type Software system CBR gt PBR PBR gt CBR CBR PBR
All defects included Seminar 7631557968 0 0
All defects included Hospital 7233873848 0 0
Syntactic defect omitted Seminar 6968160000 0 0
Syntactic defect omitted Hospital 85122000 0 0
21
Experiment results

• Individual inspector results
• Inspectors who use PBR spend on average 18 less
  time on inspection
• Cost per defect of Inspectors who use PBR is on
  average 39 higher
• There is no statistically significant difference
  in defect detection effectiveness between
  individual CBR and PBR inspectors, however on
  average
• PBR is more effective for Semantic and
  Consistency defects
• PBR is more effective for Activity and Component
  diagrams
• CBR is more effective for Syntactic defects
• CBR is more effective for Class diagrams
• Three-person virtual team results
• CBR teams exhibit higher defect detection
  effectiveness than PBR teams

22
Threats to validity

• Internal validity
• Selection experiment was a mandatory part of the
  course
• Instrumentation objects which we used could have
  influence
• External validity
• Students were used as subjects
• Design documents were similar to those which are
  used in practice, but the size of systems in
  industry is usually larger
• There were threats to internal and external
  validity, but they were not considered large in
  this experiment

23
Conclusions and Further research

• UML diagram inspection experiment was conducted
• The results of the experiment indicate that
• There is no statistically significant difference
  in effectiveness of CBR and PBR individual
  inspectors
• CBR virtual teams are more effective than PBR
  virtual teams
• Future research will be directed to further
  investigation of UML diagram inspection
• A replication of experiment was conducted in July
  2002, during which real team meeting were
  performed
• More detail analysis of both individual and team
  data
• Research of Fault Content Estimation methods

24
Software Engineering Laboratory
http//iip-lab.ics.es.osaka-u.ac.jp/
25
ADDITIONAL SLIDES 26-32
26
Experiment Variables and Hypotheses

• Independent variables
• Reading technique (CBR and PBR)
• Dependent variables
• Number of defects found
• Time spent on inspection
• Defect detection effectiveness
• Cost per defect

• Null Hypotheses
• For individual inspectors
• H01 PBR Time gt CBR Time
• H02 PBR Cost lt CBR Cost
• H03 PBR Effectiveness CBR Effectiveness
• For 3-person virtual teams
• H04 PBR Effectiveness CBR Effectiveness

27
Information systems inspected

• Seminar information system
• System supports the process of a company which
  organizes seminars
• Includes activities from seminar planning to
  issuing the qualification certificate
• Designs relationships among personnel of seminar
  company, participants, lecturers, etc.
• Hospital information system
• System supports the process of a hospital
• Includes activities such as patient registration,
  medical examination, treatment, prescription of
  the medicines, etc.
• Designs the relationships among the personnel of
  a hospital, patients, pharmacists, etc.

28
Collected data

• Two types of data collected during experiment
• Defect data
• Time data

Number of inspectors Defects Defects Defects Inspection time (min) Inspection time (min)
Number of inspectors Number of detectable defects Average number of defects detected max/min Average max/min
Seminar information system PBR User 7 7 4.43 6 / 3 60.43 90 / 46
Seminar information system PBR Designer 6 6 5.00 6 / 4 65.50 80 / 51
Seminar information system PBR Implementer 6 9 6.50 9 / 5 76.67 95 / 40
Seminar information system CBR 11 15 10.55 13 / 8 74.64 90 / 62
Hospital information system PBR User 7 7 4.43 6 / 3 48.29 70 / 25
Hospital information system PBR Designer 6 6 3.83 5 / 3 59.17 73 / 30
Hospital information system PBR Implementer 6 9 6.33 7 / 5 63.30 77 / 44
Hospital information system CBR 10 15 10.50 12 / 8 70.10 94 / 60
29
CBR Checklist
CHECKLIST Locate the following diagrams Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form. CHECKLIST Locate the following diagrams Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form. CHECKLIST Locate the following diagrams Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form.
SEQUENCE DIAGRAMS SEQUENCE DIAGRAMS SEQUENCE DIAGRAMS
10. Arent there any inconsistencies among Sequence Diagram, Class Diagram and Requirements Specification? ? yes ? no
11. Are all the necessary Objects and Messages defined? ? yes ? no
12. Is every Class from Class Diagram included in any of Sequence Diagrams and vice versa? ? yes ? no
13. Is every Use Case from Use-Case Diagram implemented in at least one of Sequence Diagrams? ? yes ? no
14. Are there no redundant elements (Objects of Messages) in Sequence Diagram? ? yes ? no

30
PBR Scenario
DESIGNER SCENARIO Assume that you are the Designer of the system. The concern of the designer is to ensure that the designers needs are completely satisfied in the following documents Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designers point of view. Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form. DESIGNER SCENARIO Assume that you are the Designer of the system. The concern of the designer is to ensure that the designers needs are completely satisfied in the following documents Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designers point of view. Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form. DESIGNER SCENARIO Assume that you are the Designer of the system. The concern of the designer is to ensure that the designers needs are completely satisfied in the following documents Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designers point of view. Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form.
Step 3 Locate the Class Diagram and Sequence Diagrams Locate the Class Diagram and Sequence Diagrams
Step 3 Make a list of all Objects included in Sequence Diagrams and answer the following questions. Make a list of all Objects included in Sequence Diagrams and answer the following questions.
Step 3 3.1. Are all the Objects of Sequence Diagrams defined in Class Diagrams?
Step 3 Are all Classes of Class Diagrams defined in Sequence Diagrams?
Step 3 3.2. Are all the Messages between objects Sequence Diagram defined as Methods of the corresponding Class in Class Diagram?
Step 3 3.3. Are there no redundant elements in Sequence diagrams?
31
Defect registration form
32
Software development process

• The main steps of software development process
• Development of Use-case diagrams
• Describing system activities in Activity diagrams
• Defining static structure of the system in Class
  diagrams
• Modelling dynamic aspects of the system in
  Sequence diagrams
• Detailed description of object states in
  Statechart diagrams
• Development of the Component diagrams
• Users perspective in our experiment covered the
  second, and partially the third and the fourth
  steps
• Designers perspective covered the third and the
  fourth steps
• Implementers perspective covered the sixth step,
  and partially the third and the fourth steps