The use of a metamodel to support multiproject process measurement

1
The use of a meta-model to support multi-project
process measurement

• Alberto Colombo, Ernesto Damiani, Fulvio Frati
• University of Milan - Italy
• Sergio Oltolina, Gabriele Ruffatti
• Engineering Ingegneria Informatica - Italy
• Karl Reed
• La Trobe University - Australia

2
Introduction

• Software companies are increasingly engaged
• in multiple projects delivered on heterogeneous
  platforms
• in co-development of software systems joint
  software development projects including staff
  from partners and customers
• They have to support multiple software
  development processes guaranteeing uniform levels
  of process enactment, and product quality across
  the entire project portfolio

3
Our Work

• Provide process measurements in a joint-project,
  multi-process model business environment
• Define a complete meta-model for computing
  across-process, multiple-project metrics
• Permit monitoring of CMMI compliance
• Our research objectives
• Participate in design and implementation the Open
  Source tool Spago4Q
• Improve understanding of metrics and process
  monitoring suitable for CMMI (and similar
  regimes) compliance
• Develop frameworks and metrics for multi-process
  model/multi-project monitoring, yielding
• A common set of reliable company-wide project
  performance measures independent of the process
  used

4
Related Work

• Tools for monitoring software projects by using
  data on defect-rates, artifact metrics and
  resource utilization
• TAME (Tools for a Measurement Environment)
• SoftMan (environment for forward and reverse
  CASE)? single-process, single-project data
  collection
• Research projects and commercial tools on project
  monitoring
• work by Holkar Torabi (La Trobe University) on
  XML-based quality monitoring system
• Hakystat, a measurement framework for
  non-intrusive project metrics
• Polarion
• 6th Sense Analytics
• ? not all are supporting cross-project
  comparison and they are not metamodel driven

5
Our Meta-model Approach

• Three main requirements
• supporting across-process data collection
• supporting multiple metrics frameworks, based on
  Goal-Question-Metric (GQM) approach
• analyzing process and product data supporting
  company-wide as well as per-process views
• Requirements met by a meta-model representation
  of software process and of measurement activity
  representing
• the process(es) to be monitored
• the measurement to be taken
• assessment framework to be used
• Components have been designed to be process
  independent, allowing the same measurement
  framework to monitor different processes

6
(No Transcript)
7
CMMI-based Assessment

• Support a number of development processes,
  instantiating a single set of KPIs and metrics
  based on CMMI
• CMMI does not specify metrics to be collected,
  method of collection to exploit, interpretation
  of the measured data
• At the time of writing, there was little research
  on the use of metrics in CMMI conformance
  evaluation
• We used CMMI goals and practices as the basis to
  a complete GQM process for the evaluation of a
  software process

8
Establish CMMI-based GQM

• Our GQM approach for CMMI, used the following
  three-step procedure
• goals were defined using the CMMI assessment
  model, focusing on objectives defined in each
  process areaEx Specific Goal 1 Manage
  requirements
• we identified measurable attributes that a single
  question should measure to produce an actual
  estimate of a CMMI goalEx Manage requirements
  changes implies the monitoring of requirement
  states
• we defined metrics for each question, and a
  threshold to determine whether a metrics value
  satisfies the underlying CMMI practiceEx
  Requirement variability

9
Across-Process Metrics

• Projects undertaken by large software companies
  make multiple software process competence and
  practice essential
• compute company-wide (as opposed to
  process-specific) process and product quality
  indicators
• establish which metrics (coming from different
  projects) can be safely aggregated and which ones
  cannot due to semantic differences

10
Across-Process Metrics - (contd)

• Two ways to handle this problem
• Adopt variations of the same development process
  (e.g., different Crystal colors)
• process variations share workproducts whose
  attributes have similar semantics
• Define specific semantics annotations to identify
  which metrics can be merged to compute
  across-processes measurements
• Quality (i.e. final product defect rates), and
  productivity related metrics that can be
  meaningfully merged

11
Across-Process Metrics - (contd)

• Examples of Problematic Comparisons
• Requirements Volatility between (say) Waterfall
  and XP
• Fault detection rates
• Delivery Rates

12
Across-Process Metrics - (contd)

• Important characteristic of CMMI is. be able
  to show improvements
• The design of a metrics program which deals with
  this is non-trivial, e.g.
• negative outcomes from an improvement activity
  could indicate bad adoption practice rather than
  an inappropriate activity
• introducing temporal issues to metrics is not
  simple
• improvements may be difficult to develop for
  best of breed projects

13
Spago4Q Platform

• Spago4Q (SpagoBI for Quality) is a Free Open
  Source platform that fully adopts the proposed
  meta-model approach
• adaptable to complex organizational contexts,
  independently from the development processes,
  infrastructure tools, measurement and assessment
  frameworks
• suitable for maturity assessment, effectiveness
  of development software process and quality
  inspection of the released software
• data and measures collected with non-invasive
  techniques

14
Spago4Q Platform - (contd)
15
Spago4Q Platform - (contd)

• The metamodel approach of Spago4Q allows real
  cross-process measurement regimes
• the same measurement can be applied to projects
  following different developing paradigms
• under some circumstances, data from these
  measurements could be gathered and analyzed
  together
• Since Spago4Q is Open Source, it is possible to
  build organization-specific extractors to deal
  with particular file formats

16
Experimental Results

• Experiments were carried out at Engineering
  Ingegneria Informatica
• Seven pilot projects data from three were
  analyzed
• different development process Waterfall, EWebMo
  (ad-hoc evolutionary-iterative process), and
  Scrum
• metrics related to the requirement management
  goal, in particular requirement definition and
  tracking, testing, and issue/bug tracking phases

17
Experimental Results - (contd)
Prj 1 EWebMo Prj 2 Waterfall Prj 3 Scrum

• Data collected by Spago4Q
• More detailed graphs and dashboards could be
  produced
• KPIs for requirements variability appropriate
  for traditional process models are unlikely to
  be meaningful for agile projects

18
Conclusions

• We present how a simple meta-model approach can
  integrate quantitative analyses of different
  processes, increasing the effectiveness of
  company-wide risk detection and mitigation
  practices
• Our technique can help in identifying the areas
  for improvements, and develop a mechanism for
  assessing company-level conformance to maturity
  goals
• Future works will include the classification of
  process models into comparison classes

19
Contact Information

• Web Site
• SesarLab sesar.dti.unimi.it
• Spago4Q www.spago4q.org
• Email
• frati_at_dti.unimi.it