Empirical Evaluation

1
Empirical Evaluation

• Considerations for Experimentation

2
General Definition

• According to the logical positivist point of
  view, the experiment is the best way to conduct
  research (Whitley, 1996).
• Why?
• Rigorous
• Causal relationships
• If x then y
• AND
• If NOT x then NOT y

3
The Case for Experimentation Benefits
Limitations

• Consider an example
• Designing a website for older adults

4
Benefits

• High degree of control
• Precise measurement (related to control)
• Allows for causal judgments

High Perceptibility (WhizBang.com)
Low perceptibility (BoringStuff.com)
5
Limitations

• Artificiality
• Environment
• Methods
• Ethics

Just be comfortable and respond normally!
6
Designing a Study

• Predictors, Hypotheses, and Pitfalls

7
Dont Forget Your Question

• What is your question?
• Review the literature
• Has you question already been answered?
• Is there a debate about your question?
• How have previous studies been designed to answer
  this question?
• www.hcibib.org

8
Experimental Factors

• Define the variables
• Hypotheses
• Method
• Participants

9
Define the Variables

• Independent variables Manipulated, Cause
• Dependent variables Measured, Effect
• The dependent variable(s) needs to be affected by
  the independent variable(s).
• Control group

10
Hypotheses

• Aim of an experiment Show that by varying the
  independent variable you have caused a difference
  in the dependent variable.
• Hypothesis A variation in the independent
  variable will cause a difference in the dependent
  variable.
• Null hypothesis There is no difference in the
  dependent variable by the independent variable.

11
Example
2 (age) x 4 (site design)
DV Time to complete the task
12
Experimental Method

• Between-groups
• Each participant assigned to a different
  condition
• Adv. No learning effects
• Disadv. Need more participants
• Disadv Individual differences could bias the
  results
• Within-groups
• Participant tested on all conditions
• Adv. Fewer participants
• Disadv. Potential learning effects
• Mixed
• One variable between-groups and one variable
  within-groups

13
Participants

• Characteristics
• Number
• Consent
• Anonymity
• Risks
• Compensation
• Free will

14
Independent vs. Confound Variables

• Independent variables
• Confound variables
• Participant variables
• Procedural variables

15
Validity
Internal Validity (Experimental)
External Validity (Naturalistic)
16
Data Analysis / Statistics

• Descriptive statistics
• Inferential statistics
• Sensitivity/power
• Normal distributions
• Outliers

17
Tips

• Experimental control, not as easy as it may seem
• Plan ahead
• Create a detailed protocol be organized
• Pilot studies

18
References

• http//trochim.human.cornell.edu/kb/
• Whitley, B.E. (1996). Principles of Research in
  Behavioral Science. Mountain View, CA Mayfield