Process Modeling and Simulation Experiments for Software Engineering

1
Process Modeling and Simulation Experiments for
Software Engineering
Nancy S. Eickelmann,PhDMotorola Labs1303 E.
Algonquin Rd.Annex-2Schaumburg, IL 60196Phone
[phone]Fax [phone]Nancy.Eickelman
n_at_motorola.com
USC-CSE Octoberr 23-26, 2001
Dr. Nancy Eickelmann
2
Overview

• Process Modeling and Simulation
• Who Uses It CMM High Maturity Organizations
• How to use it for Defect Prevention
• Simulation Experiments for Software Engineering
• Internal Validity
• External Validity
• Design of Experiments

3
Process Modeling and Simulation for High
Maturity Organizations
Software Lifecycle Process
Process Performance Cost, Quality, Schedule
Project Data Process and Product
4
State of the Practice Increasing Process Maturity
Source SEI Web Site SEMA Report for March 2000
5
Level 5 KPAs Optimizing

• Defect Prevention
• Goal 1- Defect prevention activities are planned
• Goal 2- Common causes of defects are identified
• Goal 3- Common causes of defects are prioritized
  and eliminated
• Technology Change Management
• Goal 1- Incorporation of technology changes are
  planned
• Goal 2- New technologies are evaluated to
  determine their effect on quality and
  productivity
• Goal 3- Appropriate new technologies are
  transferred into practice
• Process Change Management
• Goal 1- Continuous process improvement CPI is
  planned
• Goal 2- Organization wide process improvement
• Goal 3- Standard processes are improved
  continuously

6
Defect Prevention

• Defect prevention is defined as an activity of
  continuous institutionalized learning during
  which common causes of errors in work products
  are systematically identified and process changes
  eliminating those causes are made.

7
What is Required for Defect Prevention?

• A measurement program that provides full
  lifecycle in-process visibility
• Knowledge of how and when defects by type,
  severity, and impact are introduced into the
  product
• Methods to improve the process that will result
  in defect prevention

8
From a Risk Management Perspective

• Defect prevention through risk management means
  engaging in a set of planning, controlling, and
  measuring activities that result in obviating,
  mitigating or ameliorating defect causing
  conditions.

9
Process Simulation Models

• Experimental Simulation
• Qualitative and quantitative results based on
  non-deterministic or hybrid simulation model
• mirrors a segment of the real world
• control of variables is high
• supports testing of causal hypothesis
• results can be replicated
• high internal validity
• high external validity, generalizability

10
Key Issues for Empirical Studies

• First, software engineering has a large number of
  key variables that have different degrees of
  significance depending on the process lifecycle,
  organizational maturity, degree of process
  automation, level of expertise in the domain,
  computational constraints on the product,
  required properties of the product.
• Second, the individual key variables required to
  mirror the real world context have the potential
  property of extreme variance in the set of known
  values within the same context or across multiple
  contexts. For instance, programmer productivity a
  key variable in most empirical studies has been
  documented at 101 and 251 variances in the same
  context.
• Third, software engineering domain variables, in
  combination, may create a critical mass or
  contextual threshold not present when studied in
  isolation. To identify variables that co-vary and
  have interdependent relationships statistical
  methods are applied to the data sets.
• 1986 IEEE TSE, Basili, Selby and Hutchins

11
Empirical Research Summary

• Experimental Simulation
• Qualitative and quantitative results based on
  non-deterministic or hybrid simulation model
• Math Modeling quantitative results based on a
  deterministic model

• Mirrors a segment of the real world, control of
  variables is high, supports testing of causal
  hypothesis, results can be replicated, high
  internal validity and generalizability
• Captures real world context in which to isolate
  and control variables

• Researcher bias can be introduced through
  selection of variables, parameters and
  assumptions concerning the model. Modeling
  requires high degree of analytical skill, and
  interdisciplinary knowledge
• Results are not typically generalizable to other
  populations or environmental contexts, researcher
  bias is common.

12
Factors Jeopardizing Research Internal Validity

• History - events occurring between the 1st and
  2nd measurement of the experimental variables
• Maturation - processes impacting study results
  pertaining to the passage of time, i.e., growing
  tired, growing hungry, growing older,
  undocumented reliability growth or decay
• Testing - the effects of taking a test upon the
  scores of the 2nd test
• Instrumentation - changes in the measuring
  instrument, changes in the observers or record
  keeper perceptions
• Statistical regression - group selection based on
  extreme scores
• Bias - differential selection of comparison
  groups
• Experimental mortality - loss of respondents
• Selection/Maturation interaction - confounding
  variable mistaken for dependent variable

13
Factors Jeopardizing Research External Validity
(Generalizability)

• Testing interaction or reactive effects - altered
  respondent sensitivity due to pre-test
  measurement
• Interaction effects - confounding effects from
  selection bias and experimental variable
• Reactive effects of experimental arrangements -
  obviates applicability of results to persons or
  contexts not exposed under the experimental
  setting
• Multiple treatment interference - occurs when the
  respondent pool is reused repeatedly

14
How We Assure Internal Validity
Solomon Four Group Design
15
How We Assure Internal Validity
X
X,Y,Z
M3
X,Y
M4
X,Z
Simulation Experiment Design
16
Initialization Sub-Module

• Set the initial parameters for the model
• Inputs
• Initial Defects X
• Detection Effectiveness Y
• Correction Effectiveness
• Number of Inspections
• Inspection Size Z
• Delta Size
• Resources (Moderator, Author, Librarian,
  Recorder, Inspector, Reader, Other)
• Output
• Item Out

17
Fagan Inspection Sub-Module

• Calculate duration and number of defects found
  and removed
• Inputs
• InspectedItem
• OverviewIn, ThirdHourIn
• Output
• FaganInspectionDurationOut
• OverviewDurationOut
• PlanningDurationOut
• PreparationDurationOut
• InspectionDurationOut
• ThirdHourDurationOut
• ReworkDurationOut
• FollowUpDurationOut
• MinorDefectFoundOut
• MajorDefectFoundOut
• DefectRemovalOut
• ItemOut

18
Preliminary Results

• Captures Numeric Graphical Simulation Results
• Inputs
• Selected Intermediate and Final Module Values
• Outputs
• Duration for Each Activity
• Number of Major Defects Found
• Number of Minor Defects Found
• Number of Defects Removed
• Minimum Maximum Number of Days Expended

19
What We Need for Empirical Studies in the
Software Engineering Domain

• Process simulation experiments
• Capture and replicate the variables of the real
  world environment
• variable variances are isolated and documented
• variables are studied in isolation or in
  combination to isolate and document critical
  mass effects
• the cost to replicate the multiple real world
  environments and evaluate across projects and
  organizations is much less than field studies,
  longitudinal case studies or controlled
  experiments
• we can replicate other empirical studies and
  evaluate applicability and generalizability of
  results

20

• Thank You!
• Nancy S. Eickelmann,PhDMotorola Labs1303 E.
  Algonquin Rd.Annex-2Schaumburg, IL 60196Phone
  [phone]Fax [phone]Nancy.Eickelman
  n_at_motorola.com