SE 468 Software MeasurementProject Estimation

1
SE 468 Software Measurement/Project Estimation

• Dennis Mumaugh, Instructor
• dmumaugh_at_cdm.depaul.edu
• Office Loop, Room CDM 430, X26770
• Office Hours Monday, 400-530

2
Administrivia

• Comments and feedback
• Assignment 5 download tools from different web
  site
• Interesting Paper Survey of Cost Estimation
  Tools, https//guinness.cs.stevens-tech.edu/lbern
  ste/papers/Cost_EstimationTools.doc
• Access to ACM and IEEE journals using the DePaul
  Library
• Assignment schedule
• Assignment 5 due November 16
• Assignment 1 part 2 due November 16
• Journal due November 23
• Term paper due November 23
• Exam November 19 - 24

3
Assignment 4 feedback

• See COL gt Documents for solution, including Xcel
  spreadsheet.
• Question 4 phase most effective in removing
  defects
• Is DRR measure
• How about PCE
• Question 6 were reviews and inspections
  effective?
• Probably the PCEs for these phases were
  66/61/68. This is CMMI level 4.
• Probably DRR for coding was 9.71 but DIR was
  11.51
• Testing was effective with PCEs of 100 and low
  DIR.
• Overall the DRE of 97.65 This is CMMI level 5.

4
SE 468 Class 9

• Topics
• Customer Satisfaction Surveys and other methods
• Reading
• Kan chapter 14
• See also reading list

5
Thought for the Day

• Keeping existing customers is cheaper than
  finding new ones.
• It's the Customer, Stupid.

6
Customer Satisfaction

• The ultimate validation of quality
• Product quality and customer satisfaction
  together form the total meaning of quality

7
Total Quality Management

• Aimed at long-term business success by linking
  quality with customer satisfaction
• Enhancing customer satisfaction is the bottom
  line of business success
• Customer focus is the only way to retain the
  existing customer base and to expand market share
• Five times more costly to recruit a new customer
  than it is to keep an old customer
• Dissatisfied customers tell 7-20 people about
  their bad experience while satisfied customers
  tell only 3-5

8
Dimensions of Software Quality

• Low/no bugs gt good reliability
• Useful feature set
• Ease of use
• Timely there when you need it
• Good price-performance
• Good service and support

9
Mantras

• Software quality is measured in terms of customer
  satisfaction!
• "Customer Satisfaction through Total Quality"

10
Definitions

• Client satisfactionThe client's perception that
  the service provider's performance meets or
  exceeds his or her expectations.
• Client satisfaction measurementThe assessment of
  client expectations and of the actual and
  perceived quality of service.

11
Customer Satisfaction
12
Measuring Customer Satisfaction
Apostle
100
Zone of Affection

Profitability (Satisfaction)
80
Zone of Indifference
60
Loyalty (Retention)
40
Zone of Defection
20
Terrorist
1 Very Dissatisfied
2
3 Neither Satisfied Nor Dissatisfied
4
5 Very Satisfied
Adapted from Harvard Business Review,
November/April 94
13
Customer Satisfaction Survey

• Various ways to get customer feedback
• Telephone follow-up regarding satisfaction after
  a purchase
• Customer complaint data
• Direct customer visits
• Customer advisory councils
• User conferences
• Time-honored approach is conducting customer
  satisfaction surveys that are representative of
  the entire customer base

14
Survey Data Collection

• Four common methods
• Personal face-to-face interviews
• Telephone interviews
• Mail questionnaires
• Electronic questionnaires such as a web site.

15
Personal Interviews

• Requires the interviewer (or interview team) to
  ask questions based on a pre-structured
  questionnaire and to record answers
• Primary advantage is the high degree of validity
  of the data
• Interviewer can note specific reactions and
  eliminate misunderstandings about questions

16
Major Limitations

• Costs it is the most expensive method
• If not adequately trained, the interviewer may
  deviate from the required protocol, and bias the
  data
• Interviewer must maintain neutrality, otherwise
  even a facial expression could affect the
  response
• Errors in recording responses could lead to
  erroneous results
• Can be biased
• Responders want to please the interviewer
• Less negative responses

17
Telephone Interviews

• Less expensive
• Can be monitored by the research team to ensure
  that interview procedure is followed
• Computer-aided approach can further reduce costs
  and increase efficiency
• Should be short and impersonal to maintain
  interest of interviewee
• Limitations are lack of direct observation and
  limited group of potential respondents some
  people just wont participate

18
Mail Questionnaires

• Least expensive
• Lowest response rate that introduces bias toward
  dissatisfied customers
• Questions must be carefully constructed,
  validated, and pre-tested for final use
• Requires professional knowledge and experience to
  administer the process

19
Summary of Advantages Disadvantages
(- Disadvantage Advantage - Either)
20
Implementation Process

• The following seven steps offer a practical
  approach to measuring client satisfaction
• Step 1 Determine who the clients are.
• Step 2 Determine the objectives for measuring
  client satisfaction and define the related
  information needs.
• Step 3 Develop a measurement strategy.
• Step 4 Gather, analyze and report information.
• Step 5 Use and communicate client satisfaction
  information.
• Step 6 Review the measurement practices.
• Step 7 Repeat the process.

21
What to measure

• Product characteristics
• IBM uses CUPRIMDA Capability, Usability,
  Performance, Reliability, Ease of Installation,
  Maintainability, Documentation, Service, and
  Overall
• Hewlett-Packard uses FURPS categories
  functionality, usability, reliability,
  performance, and serviceability.
• Company characteristics
• price, product availability, and ease of ordering
• Typical measurement is by asking questions and
  answering with a 5 point (Likert) scale (next
  slide).
• Important to choose questions carefully
• Important to word questions carefully

22
Customer Satisfaction

• Customer satisfaction is usually measured on a
  five-point (Likert) scale
• Very satisfied
• Satisfied
• Neutral
• Dissatisfied
• Very dissatisfied
• Any finer granularity not meaningful. See notes.

23
Problems with 5 Point Scale

• Some data are measured at the ordinal level.
  Numbers indicate the relative position of items,
  but not the magnitude of difference. One example
  is a Likert scale.
• User ratings are at best ordinal categorizations.
  
• While it is not uncommon to calculate averages or
  means for such data, doing so cannot be justified
  because in calculating averages, equal intervals
  are required to represent the same difference
  between levels of perceived quality.
• The key issues with aggregate data based on the
  kinds of rating scales commonly used online are
  as follow
• Averages should not be calculated for data of the
  kind collected. It is usually impossible to
  evaluate the reliability or validity of user
  ratings.
• Products are not compared with respect to
  explicit, let alone common, criteria.
• Only users inclined to submit a rating for a
  product do so.
• Data are not usually published in a form that
  permits evaluation of the product ratings.

24
Sampling Methods

• For large customer base, its too costly to
  survey all customers
• Need to get a representative sample
• Use scientific probability sampling methods
• Four basic types
• Simple random sampling
• Systematic sampling
• Stratified sampling
• Cluster sampling

25
Simple Random Sampling

• Sample size n is drawn from population such that
  every possible customer has an equal chance of
  being selected
• Not to be confused with convenient or accidental
  sampling where sampler randomly selects
  individuals that he/she comes across
• Each individual in the population must be listed
  once and only once
• Some mechanical procedure generates a random
  number to draw sample

26
Systematic Sampling

• Instead of using random numbers, one simply goes
  down a list taking every Kth person
• K is the ratio between the population size and
  the sample size 1/K is the sampling fraction
• For example, if you wanted to sample 500
  customers out of a population of 20,000, then K
  would be 40 start with a random number between
  1-40 and draw every 40th person

27
Systematic Sampling Bias

• Simpler than random sampling, but can introduce
  two types of bias
• The entries on the list may have been ordered so
  that a trend occurs
• The list may possess some cyclical characteristic
  that coincides with the K value
• Neither bias occurs frequently in practice

28
Stratified Sampling

• Divide the sample population into layers or
  strata based on some important variable.
• More efficient than others
• Get greater accuracy for same cost or the same
  accuracy for less cost
• By means of stratification, ensure that
  individuals in each stratum are well represented
  in the sample
• Sampling fractions within each stratum may be
  equal (proportional stratified sampling) or
  different (disproportional stratified sampling)

29
Cluster Sampling

• A cluster sample is a simple random sample in
  which each sampling unit is a cluster of elements
• Generally less efficient than simple random
  sampling, but much more cost effective
• Select clusters as heterogeneous as possible, but
  small enough to cut down expenses such as travel
• Example Company with many branch offices may
  cluster sample using branch offices as clusters
  to reduce expense of face-to-face interviews

30
Sample Size

• How large a sample is sufficient?
• Depends on the confidence level and acceptable
  margin of error
• The higher the level of confidence and the
  smaller the error margin, the larger the sample
  size and vice versa
• For each probability sampling method, specific
  formulas are available some are quite
  complicated
• The power of a sample depends on its absolute
  size
• When the sample size is up to a few thousand, it
  gives satisfactory results even if the population
  is extremely large
• See Determining Sample Size, http//edis.ifas.ufl
  .edu/PD006.

31
Examples of Sample Size

• The following table illustrates the sample sizes
  for 10,000 customers for various levels of
  confidence with both 5 and 3 margins of error
• Note that the required sample size decreases as
  customer satisfaction level increases
• This is because the larger the p value, the
  smaller its variance, p(1-p) pq
• Using a value of p.5 will yield the largest
  sample size because pq is the largest when pq

32
Sample Size by Confidence Level
Figure 14.2 sample sizes for 10,000
33
Analyzing Satisfaction Data

• The five point satisfaction scale is often used
  in customer satisfaction surveys
• 5 Very satisfied
• 4 Satisfied
• 3 Neutral
• 2 Dissatisfied
• 1 Very dissatisfied
• Data is usually summarized in terms of percent
  satisfied
• Usually present results in run charts or bar
  charts to show the satisfied

34
Example of Customer Satisfaction Chart
35
Analyzing Satisfaction Data

• While the satisfied is probably the most used
  metric, some companies choose to monitor the
  inverse, the percent not satisfied
• Includes the neutral, dissatisfied, and very
  dissatisfied
• Rationale is to ensure focus on areas that need
  improvement
• See the following example

36
Customer Dissatisfaction
37
Specific Attributes

• As the previous examples showed, it is common to
  monitor customer satisfaction with respect to
  specific attributes
• Data provides specific information for
  improvement
• Profile indicates areas of strength and weakness
• One easy mistake is to equate the areas of
  weakness with the priority of improvement

38
Overall Satisfaction

• To answer the question on the priority of
  improvement, it must be looked at within the
  broader context of overall customer satisfaction
• The correlations of satisfaction levels of
  specific attributes with overall satisfaction
  need to be examined it is the overall
  satisfaction level that affects the customers
  purchase decision
• See the following method for determining the
  priority of improvement

39
Improvement Method

• Determine the order of significance of each
  quality attribute on overall satisfaction by
  statistical modeling (such as the regression
  model)
• Plot the coefficient of each attribute from the
  model against its satisfaction level
• Use the plot to determine priority
• See following example

40
Example of Data Analysis
Figure 14.4
41
Satisfaction with Company

• Overall satisfaction with the company is a much
  broader scope than a product oriented
  satisfaction
• Overall satisfaction is attributed to a set of
  common attributes and satisfaction levels with
  specific dimensions
• Common attributes include
• Ease of doing business
• Partnership
• Responsiveness
• Knowledge of the customers business
• Customer driven company

42
Product Quality Attributes

• Technical solutions quality/reliability,
  availability, ease of use, pricing, installation,
  new technology, etc.
• Support and service flexible, accessible,
  product knowledge, etc.
• Marketing solution, central point of contact,
  information, etc.
• Administration purchasing procedure, billing
  procedure, warranty expiration notification, etc.
• Delivery on time, accurate, post-delivery
  process, etc.
• Company image technology leader, financial
  stability, executive image, etc.

43
How Good Is Good Enough

• How much customer satisfaction is good enough?
• The key lies in the relationship between customer
  satisfaction and market share
• Basic assumption is that satisfied customers
  continue to purchase products from the same
  company
• Dissatisfied customers will buy from others
• Studies indicate that you have to be better than
  your competitors
• Important to measure both your satisfaction level
  and that of your competitors

44
Customer Satisfaction Process

• Measure and monitor the overall satisfaction over
  time, your own as well as key competitors
• Perform analyses on specific satisfaction
  dimensions, quality attributes of products and
  strengths, weaknesses, prioritization, etc.
• Perform root cause analysis to identify
  inhibitors for each dimension and attribute
• Set satisfaction targets by taking competitors
  satisfaction levels into consideration
• Formulate and implement action plans based on the
  above

45
Customer Surveys or Not

• Surveys may not be the best way to measure
  customer satisfaction
• Single person at a company
• May not be impartial
• May not have the complete picture
• May not have a realistic picture of how satisfied
  customers really are
• Can we derive customer satisfaction from sources
  other than surveys?

46
Customer Expectations

• The key to customer satisfaction is to meet or
  exceed expectations
• One needs to manage the expectations
• Key aspects
• Quality
• Customers forget delays but not the quality
• Customers forget everything else if and only if
  the quality delivered is superb.
• On-time delivery
• Money price escalations irritate customers
• Cost overruns
• Issues prompt and successful resolution of
  issues
• Accommodation and cooperation how easy was it
  for the customer to ask for and get changes

47
Quality Rating

• Quality Rating (QR) for customer satisfaction
• QR (actual defect density accepted defect
  density) accepted defect density
• For a six-sigma the accepted defect density is 3
  defects per million opportunities for 5-sigma 3
  defects per 100,000 opportunities, etc.
• Can measure defect density in KLOC or FPs.
• If QR lt 0 then customer expectations have been
  fully met.
• Sometimes a single defect is a show stopper.

48
Delivery Rating

• delivery rating (DR) for customer satisfaction
• DR (actual days taken for delivery
  accepted days for delivery) accepted days for
  delivery actual days taken for delivery is the
  number of calendar days between the date of the
  purchase order and the date on which delivery was
  actually effected. accepted days for delivery,
  number of calendar days between the date of the
  purchase order and the date of delivery specified
  on the purchase order.
• If DR lt 0 then customer expectations have been
  fully met.
• Customers might forget delayed deliveries, but
  they seldom forget poor quality.

49
Price Rating

• price rating (PR), use the price agreed to
  (before taxes) on the original purchase order and
  the final amount billed. Here is the formula for
  computing customer satisfaction in this area
  PR (actual amount billed amount on the
  purchase order) amount on the purchase order
• If PR lt 0 then customer expectations have been
  fully met.

50
Issue Rating

• Issues are disconnects between vendor and
  customer
• Unclear specifications
• Lack of understanding of requirements
• Conflict or error in requirements
• Use the issue density (ID) to compute an issue
  rating (IR).
• issue density (ID)ID number of issues raised
  software size Software size can be any software
  size measure, such as LOC or FP, just be
  consistent
• issue rating (IR)IR (actual ID standard ID)
  standard ID
• If IR lt 0 then customer expectations have been
  fully met.

51
Cooperation Rating

• Cooperation involves making customer requested
  changes.
• The concern is whether the change impacts cost or
  delivery.
• cooperation rating (CR), CR (no. of change
  requests received no. of change requests
  implemented without affecting delivery date or
  price) no. of change requests received
• If CR 0 then customer expectations have been
  fully met.

52
Composite Rating

• Compute a composite rating
• Assign a weight to each factor
• Weights must sum to 1.00
• Composite rating is
• CCSR 5 - (QRw1 DRw2 PRw3 IRw4
  CRw5)
• Possible for rating to be more than 5 this
  means we exceed customer expectations.

53
Key Points

• Most common method to gauge customer satisfaction
  is to conduct representative sampling surveys
• Three major methods face-to-face interviews,
  telephone interviews, and mailed questionnaire
• Each has its advantages and disadvantages
• Need to use scientific probability sampling
  methods to obtain representative samples
• Use one of four basic methods simple random
  sampling, systematic sampling, stratified
  sampling, or cluster sampling
• Given a probability sample, the larger the
  sample, the smaller the sampling error
• Sample must be a scientific probability sample
• When analyzing and presenting customer
  satisfaction survey data, include the confidence
  level and margin for error

54
Key Points

• Satisfaction with specific quality attributes are
  useful in developing a strategy for product
  improvement, but must be looked at within the
  broader context of overall satisfaction
• Correlations of satisfaction levels of specific
  attributes with overall satisfaction need to be
  examined so improvement actions are directed to
  maximize overall satisfaction
• Need to also consider the customers satisfaction
  with the company such as marketing, order
  processing, delivery, support, and service
• One must be better than ones competitors in
  overall customer satisfaction in order to retain
  customer loyalty and to expand market share
• Customer Satisfaction is a subjective measure.
• It is a measure of perception, not reality,
  although when it comes to a happy customer,
  perception is more important than reality.
• As our products, services and processes have
  improved, the expectations of our customers have
  increased.

55
Next Class

• Topic
• Software Process Improvement Conducting
  In-Process Quality Assessments Software Process
  Improvement
• Reading
• Kan chapters 15-19
• See also reading list
• Assignments
• Assignment 5 due November 16
• Assignment 1 part 2 due November 16

56
Journal Exercises

• What are the problems with using a Likert type of
  rating? See The Biasing of Surveys in the
  reading list
• In an survey in student evaluation, a professor
  gets a 4.5 on a five point scale (average on n
  ratings). What does this tell you?
• Does defect density or time to resolution have an
  impact on Customer Satisfaction?
• Does Customer Satisfaction improve if time to
  resolve severity 1 field defects is reduced?