SE 468 Software MeasurementProject Estimation

1
SE 468 Software Measurement/Project Estimation

• Dennis Mumaugh, Instructor
• dmumaugh_at_cdm.depaul.edu
• Office Loop, Room CDM 430, X26770
• Office Hours Monday, 400-530

2
Administrivia

• Comments and feedback
• Journal
• Term paper
• Choose topic and get approval by October 5.
• Deliver end of classes (November 23)
• Suggested topics
• Function Point Analysis limitations and
  criticisms
• COCOMO how does it work? Limitations and
  enhancements.
• Statistical SQA
• Other topics of similar nature

3
SE 468 Class 2

• Topics
• Measurement
• Measurement Key Concepts
• Measurement Information Model
• The Business Decision-Making Process 
• Reading
• Kan chapter 3
• Articles on the Class Page
• FP-05 Software Measurement
• Software Measurement Key Concepts and
  Practices,  John McGarry, Nov 9, 2001
• Measurement Information Model, John McGarry, Nov
  9, 2001
• Defining and Understanding Software Measurement
  Data, James A. Rozum, CMU/SEI

4
Thought for the Day

• "What gets measured gets managed."
• Business Proverb
• "If you can not measure it, you can not improve
  it."
• Lord Kelvin

5
Measurement Theory

• The relationships among theoretical concepts,
  definitions, and measurement

6
Measurement and Data

• Drives the progress of science and engineering
• Without empirical verification, theories
  propositions remain abstract
• See example proposition that the more rigorously
  the front end of the software development process
  is executed, the better the quality at the back
  end
• From concepts to measurement, there are several
  steps with different levels of abstraction

7
Theory

• Consists of one or more propositional statements
  that describe the relationships among concepts
• Usually expressed in terms of cause and effect
• From each proposition, one or more empirical
  hypotheses can be derived
• Data is collected and analyzed to prove or
  disprove the hypotheses

8
Abstraction Hierarchy
9
Theoretical Definition

• The building blocks of theory are concepts and
  definitions
• In a theoretical definition, a concept is defined
  in terms of other concepts that are already well
  understood
• In a deductive logic system, certain concepts
  would be taken as undefined primitives
• All other concepts would be defined in terms of
  the primitive concepts
• Example
• The concepts point and line may be used as
  undefined and the concepts of triangle or
  rectangle can then be defined based on these
  primitives
• A triangle is three lines joined together at
  their beginning and end points
• Whats a rectangle?

10
Operational Definition

• In contrast to theoretical definitions, they
  actually spell out the metrics used and the
  procedures to be used to obtain data
• Example An operational definition of software
  defect rate
• the formula for the defect rate
• what defect is to be measured (numerator)
• what denominator (LOC, Function point) is used
• how to measure, etc.

11
Level of Measurement

• There are four levels of measurement
• Nominal Scale
• Ordinal Scale
• Interval Scale
• Ratio Scale

12
Nominal Scale

• The most simple operation in science and the
  lowest level of measurement is classification
• Sort elements into categories with respect to a
  certain attribute
• For example, could classify software products by
  product type such as System, Business, or Other
• In a nominal scale, the two key requirements are
  that of jointly exhaustive and mutually exclusive
• What do these terms mean?

13
Nominal Scale

• Jointly exhaustive all categories together
  cover all possible categories of the attribute
• other is frequently needed to make the
  categories jointly exhaustive
• Mutually exclusive a subject can be classified
  into one and only one category
• The names of categories their sequence bear no
  assumptions about relationships
• Example?
• Deck of Cards
• Players on a football team
• Software Development Phases
• Software defects by phase of origin

14
Ordinal Scale

• Refers to the measurement operations through
  which the subjects can be compared in order
• May classify SDP according to SEI maturity
  levels Level 1, Level 2, etc.
• Higher than the nominal scale because in addition
  to grouping, we can order the categories
• Asymmetric if AgtB is true, then BgtA is false
• Transitive if AgtB and BgtC, then AgtC
• However, there is no information about the
  magnitude of the difference between categories

15
Ordinal Scale

• Using the five point Likert scale (1completely
  dissatisfied, 2somewhat dissatisfied, 3
  neutral, 4 satisfied, 5completely satisfied)
  for customer satisfaction data, we know only that
  5 is more satisfied than 4 cant say how much
  more satisfied
• When translating order relations into
  mathematical operations, we should not use
  operations like addition, subtraction, etc.
• In real world, its assumed that theyre equally
  spaced and averages are taken
• Lets develop some examples
• Deck of Cards
• Best players on a football team by position
• Sequence of Software Development Phases
• Cost of fixing software defects by phase of
  origin

16
Interval Scale

• Can indicate exact differences between points
• The mathematical operations of addition and
  subtraction can be applied to interval scale data
• Need a well defined unit of measure that can be
  agreed to as a common standard and is repeatable
• Example if product As defect rate is 5.0
  defects per LOC, product Bs is 3.5, product C
  is 2.0, then we can say that the difference
  between product A B is the same as the
  difference between B C
• Lets develop some examples
• Blackjack value of a deck of Cards
• Number of interceptions for each quarterback
• Average time to complete each Software
  Development Phase
• Average cost to correct software defects by phase
  of origin

17
Ratio Scale

• When an absolute or non-arbitrary zero point can
  be located in an interval scale, then it becomes
  a ratio scale
• Highest measurement and all mathematical
  operations can be applied including /
• Almost all interval scales are also ratio scales
• Finally, measurement scales are hierarchical
  each higher order scale possesses all properties
  of the lower ones
• Lets develop some examples
• In Blackjack, the ratio of 10 point cards to one
  point cards
• Ratio of linemen to running backs on a football
  team
• Ratio of software defects by phase of origin to
  total defects

18
Use Highest Level

• The higher level of measurement, the more
  powerful analysis can be applied to the data
• So we should devise metrics that can take
  advantage of the highest level of measurement
• For example, in defect measurement we can always
  make various types of comparisons if the scale is
  in terms of actual defect rate - if in terms of
  excellent, good, average, etc. then our ability
  to perform additional analysis is limited

19
Some Basic Measures

• Regardless of the measurement scale, when we have
  data, we need to analyze it to understand what it
  means
• Lets look at four basic measures that are
  seemingly easy, but often misused
• Ratio
• Proportion
• Percentage
• Rate

20
Ratio

• Result of dividing one quantity by another
• The numerator and denominator are from two
  distinct populations and are mutually exclusive
• Example Test to development headcount ratio
  would be obtained by dividing the of people in
  the test organization by the of developers
• If the number is small (lt1), then the developers
  are probably doing extensive development tests

21
Proportion

• Different from ratio in that the numerator is
  part of the denominator p a/(ab)
• Ratio is used for two groups while proportion is
  used for multiple categories in one group
• Example abc N then a/Nb/Nc/N 1
• The numerator and denominator in a proportion
  need not be integers, but when they are units in
  a continuous scale than they are called fractions
• Example time in days (5.5 days)

22
Percentage

• A proportion or fraction becomes a percentage
  when it is expressed in terms of hundred units
  (denominator is normalized to one hundred)
• Frequently used in reporting results and also
  frequently misused because percentages represent
  relative frequencies based on a number of cases
• If the number of cases is small, then the
  percentages may be misleading
• Use absolute numbers when the number of cases is
  less than 30

23
Example of Table
24
Another way to look at the same data
25
Rate

• Ratios, proportions, percentages are static
  summary measures while rate is a dynamic measure
  generally defined as a measure of change in one
  quantity (y) per unit of another (x) on which the
  former (y) depends
• Usually the x variable is time and it should
  always be specified when describing a rate of
  change with respect to time
• Also, includes the concept of exposure to risk
  Kan pp. 65-66
• All elements or subjects in the denominator have
  to be at risk of becoming or producing the
  elements or subjects in the numerator
• Example of release 2 vs. release 3
• Release 2 was in the field for only two months
  while release 3 was in the field for two years
• Release 2 had 50 customer reported defects while
  release 3 had 330
• Which was the better release?
• In terms of defects per month of customer
  exposure, Release 2 was 25 while release 3 was
  1.375 defects per customer month
• How is this possible?

26
Some famous words from Aristotle
It is the mark of an instructed mind to rest
satisfied with the degree of precision which the
nature of a subject admits, and not to seek
exactness when only approximation of the truth is
possible.
Aristotle(384-322 B.C.)
27
Precision and Accuracy Discussion

• Significance of measurements
• What do we mean when we say that two points are
  3.375 apart?
• 3.375 means 3.375 0.0005
• What do we mean when we say that two points are 3
  ? apart?
• 3 ? means 3 ? ?16
• Some people call this precision but it really is
  called significance, as in significant figures
• Example
• 5 inches vs. 5.0000 inches
• Most engineering groups have the rule that
• Measurements expresses in fractions are to the
  nearest division of the scale (ruler), e.g. ?16
  or to one-half the last significant place (e.g.
  ?)
• Measurements in decimal are to nearest one-half
  of the last significant place 5 inches means 5
  0.5 inches

28
Precision, accuracy, reliability and repeatability

• Accuracy is the degree of veracity while
  precision is the degree of reproducibility.
• Accuracy is the degree of closeness of a measured
  or calculated quantity to its actual (true)
  value.
• Precision
• The ability of a measurement to be consistently
  reproduced.
• The number of significant digits to which a value
  has been reliably measured. really significance
• Many confuse precision with significance
• Accuracy is closely related to precision, also
  called reproducibility or repeatability, the
  degree to which further measurements or
  calculations show the same or similar results
• The results of calculations or a measurement can
  be accurate but not precise, precise but not
  accurate, neither, or both.
• A measurement system or computational method is
  called valid if it is both accurate and precise.

29
Precision and Accuracy
30
Reliability

• Refers to the consistency of a number of
  measurements taken using the same method
• If repeated measurements are highly consistent,
  then reliability is high
• While measurements always contain some chance for
  error, the goal is to achieve the best possible
  reliability
• Index of variation is a ratio of standard
  deviation to mean called IV the smaller IV, the
  more reliable the measurements

31
Validity

• Refers to whether the measurement or metric
  really measures what we intend to measure
• Measurements that are reliable may not
  necessarily be valid
• Example a new bathroom scale may give reliable
  measurements, but if the scale is consistently
  off by 10 pounds (not zeroed) then it is not
  valid
• It is often hard to recognize that a certain
  metric is invalid and even more difficult to
  improve it

32
Validity and Reliability

• For data to be reliable, the measurements must be
  specifically defined which increases the risk of
  not being able to represent the theoretical
  concept
• Validity is associated with systematic error that
  we can reduce through a better understanding of
  the concept that we try to measure
• Reliability is associated with random error that
  we can reduce by good measurement operational
  definitions and thus better execution of
  measurement operations and data collection

33
Correlation

• The most widely used statistical method in
  assessing relationships among observational data
• Caution must be exercised when using it
• Most of the time when one mentions correlation,
  it means linear correlation so if the correlation
  coefficient between two variables is weak, it
  usually means that there is no linear
  relationship
• But there may be a non-linear correlation so it
  is best to look at scatter diagrams to see if
  there is a particular type of nonlinear
  relationship
• The least-squares method of linear correlation is
  very vulnerable to extreme values so again use
  scatter diagrams to look at the data and detect
  the presence of a few extreme outliers

34
Causality

• In addition to empirical correlation, the
  relationship has to be examined in terms of
  sequence of occurrence or deductive logic
• In other words, does the cause precede the
  effect in time or as shown clearly in logic
• Finally, you have to determine that the
  correlation is not due to some spurious
  relationship
• Remember Correlation does not imply causation
• This is more difficult in observational data than
  in experimental data

35
Key Points

• There are four levels of measurement nominal
  scale, ordinal scale, interval scale, ratio
  scale
• Measurement scales are hierarchical each higher
  order scale possesses all properties of the lower
  ones
• Devise metrics that can take advantage of the
  highest level of measurement
• A ratio is the result of dividing one quantity by
  another the numerator and denominator are from
  two distinct populations and are mutually
  exclusive

36
Key Points

• A proportion is different from ratio in that the
  numerator is part of the denominator p a/(ab)
• Ratio is used for two groups while proportion is
  used for multiple categories in one group
• A proportion or fraction becomes a percentage
  when it is expressed in terms of hundred units
• Ratios, proportions, percentages are static
  while rate is a dynamic measure a measure of
  change in one quantity (y) per unit of another
  (x)

37
Key Points

• Reliability refers to the consistency of a number
  of measurements taken using the same method
• Validity refers to whether the measurement or
  metric really measures what we intend to measure
• Correlation is the most widely used statistical
  method in assessing relationships among
  observational data, but caution must be exercised
  when using it
• Empirical correlation has to be examined to
  determine if the cause precedes the effect in
  time or as shown clearly in logic

38
Measurement Key Concepts and Practices
39
Measurement

• All successful software organizations implement
  measurement as a part of their day-to-day
  management and technical activities.
• Evolved into a key software engineering
  discipline
• Now considered to be a basic software engineering
  practice as evidenced by its inclusion in the
  Level 2 maturity requirements of the SEIs CMMI
  products and related commercial software process
  standards

40
Measurement Culture

• Provides the objective information needed to make
  informed decisions that positively impact
  business and engineering performance
• Measurement-derived information is an important
  resource and is made available to decision makers
  throughout all levels of management
• It isnt just another thing to do, but is part
  of the engineering culture
• Managers consistently ask show me the data

41
Implementation

• Overall value of measurement is related to how it
  is implemented and used
• Most effective when implemented in support of an
  organizations business and technical objectives
• Must be integrated with existing technical and
  management activities that define a software
  project
• How should it be related to the risks and
  problems that may impact a project?

42
Measurement Life Cycle

• Measurement data and analysis results support
  both short and long-term decisions
• Typically use data to
• Plan and evaluate a proposed project
• Objectively track actual project progress
• Guide software process improvement decisions
• Help assess overall project performance
• Measurement is used during what part of the
  software development life cycle?

43
Motivation for Measurement

• Software has become a major factor in corporate
  investment and business strategies for all
  organizations
• SW is a key component in an organizations
  ability to remain competitive in a rapidly
  changing business environment
• At the project level
• Helps project manager to do a better job by
• Supporting more realistic estimates
• Better allocation of resources (schedules)
• Tracking and controlling project performance
  against plans
• Integrating project data with other technical
  management disciplines

44
Motivation for Measurement

• Allows the software project manager to make
  decisions using objective information to
• Communicate effectively
• Track specific project objectives
• Identify and correct problems early
• Make key trade-off decisions
• Justify decisions
• Provides Objective Information for Decision
  Makers

45
Organizational Discriminator

• To be a top performer in todays business
  environment
• An organization needs the right kind of
  information, on a regular basis, to make the
  right decisions
• Must use information to become more efficient and
  produce better quality products
• Have to learn and adapt in a rapidly changing
  marketplace that is extremely competitive
• Measurement
• Facilitates and accelerates organizational
  learning
• Supports corporate adaptation within the
  marketplace
• Provides a structure for learning from each
  project (good and bad)
• Helps to understand the gaps between its
  performance and what is needed by the marketplace

46
Project Management

• Foundation of any organizational software
  measurement program is established at the project
  level
• Performance is based on the success of its
  projects
• Project level decisions determine the success of
  most projects
• In todays competitive environment, there is
  little room for project rework and restarts
• Project Manager Must make daily decisions on
  how the technical product will be developed and
  to manage resources
• Involves prioritization
• More effective the decisions
• More objective data
• Planned objectives
• Established constraints

47
Project Management

• Some questions that may need to be answered
• How much effort did this project require?
• Did the project adhere to its schedule?
• What did the team produce?
• How good is the product?
• What unexpected problems arose?
• Other questions specific to your project.
• The organizational measurement approach must be
  adaptable to effectively address the unique
  information needs of each project
• Practical Software Measurement (PSM) see reading
  list focuses on project-level measurement and
  shows how measurement can be tailored to satisfy
  the needs of various projects

48
Beyond Projects

• There are valid information needs beyond project
  management
• PSM recognizes these needs, but in almost all
  cases the objective organizational-level
  information is derived from projects
• Organizational software measurement can be viewed
  as measurement across many projects
• Most corporate measurement activities take place
  at the project level

49
What Makes Measurement Work

• Two key characteristics of a successful
  measurement program
• Measurement data relates directly to the
  information needs of the project decision makers
• There is a structured and repeatable measurement
  process that is flexible and adaptable to support
  existing software development processes and
  environments
• Integrated Measurement Process
• To be effective, measurement must be implemented
  within a project as a supporting software
  engineering process
• It does not stand alone!
• It is implemented within the project environment
  to define what information the decision makers
  need and how this information is collected,
  analyzed, presented, and used

50
Measurement Information Model

• Fundamental concept inherent to a successful,
  information-driven measurement program
• A mechanism for linking defined information needs
  to the project software processes and products
• Provides a structure that defines specific
  project measures and relates them to the needs of
  project decision makers
• It directly supports both measurement planning
  and analysis activities

51
Measurement Information Model
52
Measurement Process Model

• Works in conjunction with the Measurement
  Information Model to provide an application
  framework for implementing measurement on a
  project
• Both models work together to define a measurement
  program appropriate for each particular, and
  unique, project

53
Measurement Process Model
54
Measurement Process Model

• Built around a typical Plan-Do-Check-Act
  management sequence
• Four primary activities
• Plan Measurement
• Perform Measurement
• Evaluate Measurement
• Establish and Sustain Commitment

55
Practical Software Measurement

• Addresses the development of a project
  measurement information structure using the
  Measurement Information Model
• Describes measurement activities and tasks using
  the Measurement Process Model

56
The Business Decision-Making Process
57
Software Business Case

• An assessment of the economic justification for a
  software-based system project

58
Organizational payoffs

• An embedded software project must make money for
  the company to be considered successful
• Project success as seen by the organizations
  bean counters is positive cash flow
• This is often not a motivation for software
  engineers as they usually dont see more money in
  their pockets whether the project is a financial
  success or failure
• Over time, failed projects will result in lost
  jobs!

59
Elements of Cost Analysis

• Requires a reasonable amount of detail, but not a
  very detailed estimation of cost by phase or
  activities
• In addition to the new system implementation
  costs, it must include any start-up equipment
  costs as well as user training on the new system
• Other costs that might be included are conversion
  costs (converting to a new database) and user
  costs to support the new system (people to work
  with the development team both during development
  and to help resolve problems during operation)

60
Other Possible Costs

• The following is a short list of some other
  possible system costs
• Clerical costs
• Related personnel costs like overtime, hiring,
  relocation or termination
• Related computing costs such as installation,
  maintenance, software licenses, terminals,
  telephones, servers, etc.
• Supplies costs like disks, forms, paper,
  printers, etc.
• Telecommunications costs such as line charges,
  modems, multiplexers, cables, connectors, etc.
• Facility costs like electricity, HVAC, security,
  RE taxes, etc.

61
Elements of Benefit Analysis

• Usually broken down into two classes of benefits
• Tangible Benefits direct cost savings or
  increases in revenue
• Intangible Benefits less quantifiable, but
  frequently more important such as improved
  quality of service, improved morale, increased
  growth potential, etc.
• If there are no serious intangible problems
  created by the new system (such as significant
  loss of information security), then it is best to
  try to justify the new system based on tangible
  benefits alone
• This usually results in a less subjective decision

62
Tangible Benefits

• Depend a great deal on the functionality of the
  new system, but here are two broad categories
• Increased revenues how will the new system make
  the company more profitable. This may include
  market projections based on several of the
  non-tangible benefits. Sometimes you have to be
  very creative.
• Reduced costs such as less inventory, less
  clerical support, reduced software maintenance
  cost, reduced computing costs, etc. These are
  frequently over estimated because there are many
  new costs to support the new system that arent
  recognized in the early stages of analysis

63
Be Conservative

• Its usually a good idea to perform the initial
  cost-benefit analysis using some conservative
  simplifying assumptions
• These make the analysis easier and if based on
  the conservative numbers, you can justify the
  system, then youll avoid the complications of a
  more involved analysis
• Because project managers often deal with business
  executives, they must understand how to speak
  their language

64
Business Case

• Materials prepared for decision makers to show
  that the idea being considered is a good one and
  that the numbers that surround it make financial
  sense
• Good starting point is the GQM paradigm Basili,
  1988
• Identify relevant goals
• List pertinent questions
• Identify metrics to be used to measure cost
  benefits

65
GQM Example
66
Financial Analysis of Projects

• Financial considerations are often an important
  aspect of the project selection process.
• Three primary methods for determining the
  projected financial value of projects
• Net present value (NPV) analysis
• Return on investment (ROI)
• Payback analysis

67
The Business Environment

• Business value is as important as technical value
• Business value (in relationship to technical
  value) must be quantified
• A common approach return on investment (ROI)
  what is given up for other purposes
• ROI is interpreted in different terms reducing
  costs, predicting savings, improving
  productivity, and costs (efforts and resources)

68
Return on Investment

• Return on investment (ROI) is calculated by
  subtracting the project costs from the benefits
  and then dividing by the costs.
• ROI (total discounted benefits - total
  discounted costs) / discounted costs
• The higher the ROI, the better.
• Many organizations have a required rate of return
  or minimum acceptable rate of return on
  investment for projects.
• Internal rate of return (IRR) can by calculated
  by setting the NPV to zero.

69
Payback Calculation

• The payback calculation is a lot like a mortgage
  loan amortization schedule
• Allocate all of the system costs over the life of
  the system against the quantifiable benefits that
  the organization will accrue over the life of the
  system
• Its important that you show this against a time
  line so that the client understands how much
  capital will be required to fund the project

70
Payback Period

• The payback period is the amount of time it will
  take to recoup, in the form of net cash inflows,
  the total dollars invested in a project.
• Payback occurs when the cumulative discounted
  benefits and costs are greater than zero.
• Many organizations want IT projects to have a
  fairly short payback period.

71
Cost Benefit Analysis Example

• Heres a very simple example for a system with an
  estimated development schedule of 6 months
• The initial system costs are
• Development (2 25,000 payments) 50,000
  
• Equipment misc. expenses 15,000
• Monthly operating expenses 2,000
• The benefits are
• Increased profits3,000 per month for 1st 6
  months
• 6,000 per month for
  next 6 months
• 9,000 per month for
  next 12 months
• Reduced costs 1,000 per month

72
Payback Period

• One final statement that can be made from the
  payback curve is how long it will take for the
  client to get back his or her investment
• Its kind of like When will I payoff my
  mortgage?
• In our example, its 13 months after the start of
  development or 7 months after the system goes
  into operation

73
Reference

• Practical Software Measurement (PSM) is covered
  in the reading list as
• Practical Software and Systems Measurement A
  Foundation for Objective Project Management, v.
  4.0b1 (PDF) Registration  required. or see
  (Mirror)
• And as a book
• John McGarry, David Card, Cheryl Jones, Beth
  Layman, Elizabeth Clark, Joseph Dean, Fred Hall, 
  Practical Software Measurement Objective
  Information for Decision Makers, Addison-Wesley
  2002, ISBN 0-201-71516-3
• Chapters one and two are in the reading list for
  this lecture.

74
Next Class

• Topic
• Size
• Measuring Software Size
• Estimating Software Size
• Plan Measurement
• Size Effort Estimation models
• COCOMO
• Reading
• Kan pp. 56, 88-91, 93-96, 456
• Articles on the Class Page
• Assignment 1
• Due September 28, 2009

75
Assignment 1

• Due September 28, 2009.
• Review the GQM paper and develop an exercise
  using this.
• Purpose  To get the student to see how metrics
  can help to reach a goal.
• Start recording metric data that will be recorded
  throughout the class and used for a follow up
  report.
• Possible Goals
• Improve personal software process
• Improve work/family balance
• Improve health (e.g. exercise)
• Improve personal time management (e.g. control
  email)
• Improve something that really means a lot to you

76
Journal Exercises

• As a young engineer I wrote an instruction
  Place a hole approximately 3.375 inches from the
  edge. My boss had a long chat with me about it.
  What is wrong with the instruction ?
• When counting things, such as red and green MMs,
  what can be said about the results? How accurate
  is 1257 red, and 1783 green. Is the number
  reproducible? Consider the 2008 election results
  for Senator in Minnesota.