An Experimental Validation of MetricView Visualizations

1
An Experimental Validation ofMetricView
Visualizations

• Christian Lange
• Michel Chaudron

??
EmpAnADa ProjectEmpirical Analysis of
Architecture and Design Quality
2
Outline

• About MetricView Evolution
• Purpose of the Experiment
• Design of the Experiment
• Results
• Conclusions Future Work
• Discussion

3
About MetricView Evolution

• UML models are becoming central artefacts
• ? Larger variety of activities performed on
  models
• Can UML and existing tools deals with these
  activities?
• Traceability
• Which use cases require class X?
• In which Sequence Diagrams is class X
  instantiated?
• Metrics
• Why has class X a coupling metric of 33?
• External Data
• Which class contains the most bugs?
• Evolution
• How did the quality of class X evolve over the
  past versions?

4
Example Views

• Demo
• MetricView Evolution

5
Purpose of the Experiment

• Purpose
• Validation of MetricView visualizations
• for comprehension of UML models
• Research Questions
• RQ 1 Is the correctness improved by using MV?
• RQ 2 Is the effort reduced by using MV?
• Focus on
• Structural Visualizations (Navigation)
• Metrics Visualization (Quality Assessment)

6
Experimental Task

• Task Subjects had to answer a questionnaire
• about a UML model using a tool

7
Questionnaire

• Task Subjects had to answer a questionnaire
• about a UML model using a tool
• Multiple-choice
• 29 questions
• 4 categories of questions
• Structure within class diagram(s)
• Interactions (SD CD relations)
• Functionality (UC CD SD relations)
• Metrics

8
Questionnaire II (Examples)

• Category 1 (Class Structure)
• To which classes is the class Obstruction_DB
  coupled by associations?
• Category 2 (Interaction)
• Which classes call methods of class
  CurrentLocation?
• Category 3 (Functionality)
• Which classes contribute to implementing the use
  case Leave Route?
• Category 4 (Metrics)
• In which class diagram does the class with the
  largest Number of Attributes occur?

9
Experimental Objects

• Task Subjects had to answer a questionnaire
• about a UML model using a tool
• 2 different models
• Application Domain
• Insurance Information System
• Car Navigation System
• 39 classes
• 5 class diagrams
• 5 sequence diagrams
• 11 use cases

10
Experimental Treatment

• Task Subjects had to answer a questionnaire
• about a UML model using a tool
• Treatment
• MetricView Evolution
• Control Group
• Poseidon UML SDMetrics

11
Subjects

• Task Subjects had to answer a questionnaire
• about a UML model using a tool
• 100 subjects
• MSc students in the course Software
  Architecting
• Sufficient background in UML

12
Operation

• Preparation
• Lecture
• 2 assignments
• Pilot run
• Design

• Details
• Randomization
• Exam-like setting
• Debriefing questionnaire after each run

MetricView Evolution
Poseidon SDMetrics
Group A
Group B
First Run Insurance Model
Group B
Group A
Second Run Car Navigation Model
13
Results Correctness
1.0

• Variable
• Test
• Mann-Whitney
• Student t

Correct answers
0.9
Questions
0.8
0.7
0.6
First Run
Second Run
0.5
MVE PoS MVE PoS
Mean .837 .771 .921 .881
Difference 8.5 8.5 4.5 4.5
P-value lt0.001 lt0.001 0.002 0.002
14
Results Effort
120

• Variable
• Time (minutes)
• Test
• Mann-Whitney
• Student t

100
80
60
40
20
First Run
Second Run
0
MVE PoS MVE PoS
Mean 54.6 67.6 52.5 66.0
Difference -19.2 -19.2 -20.6 -20.6
P-value lt0.001 lt0.001 lt0.001 lt0.001
15
Debriefing Questionnaire

• No differences between groups
• Skill knowledge
• Motivation
• No differences between models/runs
• Quality, understandability, completeness
• Differences between runs for Correctness
• Learning effects
• Tool
• Task

16
Tool Preference
17
Conclusions Future Work

• MetricView visualizations improve Model
  Comprehension
• Effort (20)
• Correctness (4 - 8)
• Subjective Evaluation Results ? low Adoption
  Threshold
• Future work
• Investigate Learning Effects
• Variations in Comprehension Task Change Task
• Further Improvement of Visualizations

18
Towards Task-Oriented Modeling using UML
Discussion

• www.win.tue.nl/empanada

??
EmpAnADa ProjectEmpirical Analysis of
Architecture and Design Quality
19
(No Transcript)
20
Usefulness // Correctly
21
Usefulness // Efficiently
22
Prototype Metrics over Time
2 prototype implementations
23
Quality Tree

• Tailorable
• Aggregate metrics

24
Timeline

• Visualizes the change of Metric or Quality
  Attribute values over Time
• Also usable for higher abstraction levels
• Combines intuitive concepts of calendar and graph

25
Meta-Diagram

• System Overview
• Inter-diagram relations

26
Context Diagram

• Complete model context, not just one diagram

27
Tasks

• Program Understanding
• Navigation
• Model Development
• Editing
• Maintenance
• Quality Evaluation
• Testing

28
Properties

• Direct
• Internal
• Indirect
• Internal
• External

• Methods
• Name
• Number of methods

• Metrics
• Number of instantiations of a class
• Use cases related to a class

• Lines of Code
• Evolution data
• of bugs

Function draw() asdfasf asfsf adfasdfasdfa.
fpr afasdf asdfsafd Function
draw() adfasdfasdfa. fpr afasdf asdfsafd
CVS
Bugs
29
Views

• Visualize
• Required Property
• Mapped to ModelElement
• Visual Attributes to be used
• Layout
• Size
• Color
• Shape
• Orientation