Witch Doctors and Sugar Pills

1
Witch Doctors and Sugar Pills

• Barbara Kitchenham

2
Subject Experimenter Impacts

• Human-centered disciplines subject to
  experimenter subject induced distortions
• Accepted in medicine, sociology, psychology etc.
• Placebos cure illnesses
• Patients get better because doctors say they will
• Virtually ignored in Software Engineering
• Although cause of problems
• Criticism of experiment on formal methods
• Published in TSE
• Identified motivation effects as a serious
  problem for the validity of the experiment

3
Distortion Bias

• Distortion
• Additional variance due to impact of experimenter
  and subject behavior and expectations
• May not be able to detect difference among
  treatments
• May underestimate importance of differences
• Bias
• Systematic distortion
• Distortions that affect treatments differentially
• May lead to completely wrong results

4
Standard Theory 1/2

• Rosnow and Rosenthal
• Categorise experiment bias
• Observer Bias
• We see what we expect to see
• Interpreter Bias
• We code/mark results in favor of our own
  preconceptions
• Intentional Bias
• We might even fudge the results
• Glass Some people, I think, are making up
  numbers
• Conte et al. Indeed the smooth exponential drop
  in productivity exhibited in Table 5.18 and the
  model in Equation (5.14) give us some cause for
  concern. It is simply too smooth to represent
  real-world projects

5
Standard Theory 2/2

• Experimental artifacts that cause distortions
• Biosocial, Psychological, Situational, Modeling
  effects
• Effects due to attributes of the experimenters,
  interactions with subjects and the research
  setting
• Experimenter Expectancy Bias
• Have a real problem if we evaluate our own
  technologies
• Have a responsibility to evaluate our own work
• Cannot avoid possibility of bias

6
Experimenter expectancy

• Rosnow Rosenthal suggest 6 approaches to
  minimise this problem
• Most important are
• Replication
• Blinding when ever possible
• Reducing contact between subject experimenters

7
Replication

• Cannot afford replication to be too exact
• Source of systematic bias can be replicated
• Good replication practice
• Use the same hypotheses
• Use different experimental protocols
• Parallel v. sequential v. mixed
• Use different materials and tasks
• Artificial v. Industrially derived
• Use different subject types
• Students v. Practitioners
• Undertaken by independent research groups
• See Miller, 2004

8
Assist Replication

• Identify the real validity issues
• Problems not resolved by experimental protocol
• Remaining limitations e.g.
• Are the treatment and control conditions
  appropriate?
• Represent the way that the technique is used/will
  be used in industry
• Are other treatment conditions important?
• Ensure that infrastructure issues are clearly
  reported
• Are the training requirements and pre-requisites
  clear?
• Are any necessary tools/ development environments
  available to other researchers?

9
Discussion Issues

• How do we improve experimental protocols to
  address subject-experimenter expectations?
• How do we improve experimental reporting to
  assist replication?
• How do we make replication a part of our research
  culture?
• How do we perform high quality replication
  studies?
• What are the most appropriate protocols?
• What other disciplines should we look to for
  advice?

10
References

• Conte, S.D., Dunsmore, H.E. and Shen, V.Y.
  Software Engineering Metrics and Models.
  Benjamin/Cummings Publishing Company Inc., 1986.
• Robert L. Glass. Good numbers And bad. HSS 44,
  1998, pp85-86.
• Ralph L. Rosnow and Robert Rosenthal. People
  Studying People Artifacts and Ethics in
  Behavioural Research. W.H. Freeman Co, New
  York, 1997.
• James Miller. Replicating software engineering
  experiments a poisoned chalice or the Holy
  Grail  Information and Software Technology,
  Volume 47, Issue 4, 15 March 2005, Pages 233-244