Experimental Methodology

1
Experimental Methodology

• Maggie De Young
• Julie Harris
• Heather McCullough
• Ryn Nasser
• Wayne Stone

2
Basic Terminology

• Experiment
• Experimental groups
• Control group
• Experimental Subject
• Treatment
• Control

3
Basic Terminology

• Field experiments
• Laboratory experiments
• Pretest/posttest
• Randomization
• Internal validity
• Reproducibility

4
Four Cell Experimental Design
Value of DV after treatment
Value of DV prior to treatment
Control Group
Experimental Group
5
Other design issues/concepts

• Random number tables used to ensure
  randomization of subject assignment to groups
• Factorial designs experimental designs that
  involve two or more independent variables

6
Evaluating the Experiment

• Null hypothesis a statement that no significant
  difference exists between the control and
  experimental groups.
• The evaluation is conducted using t-tests,
  chi-square tests, and analysis of variances
  (ANOVA) tests.

7
Sensitivity

• Sensitivity experiments ability to detect
  relatively small effects.
• Increase sensitivity by increasing the number of
  subjects (this decreases the chance for
  random/experimental error)

8
External Validity

• External validity the extent to which a sample
  is representative of the target population
• Can the results be generalized?
• If so, what is the population to which it can be
  generalized?
• Can the results be generalized to the real world?
• Easier to justify generalizations for field
  experiments than laboratory experiments because
  lab experiments are so artificial

9
Ex Post Facto Study

• Ex post facto Latin for after the fact.
  Example of a quasi-experimental design
• Quasi-experimental design Not a true
  experimental design because it lacks control, but
  it seeks to analyze an event that has already
  happened in an effort to isolate the causal
  relationships of the event

10
Considerations with experiments

• Most appropriate technique to test hypotheses
  which involve causal relationships because of the
  control mechanisms built into the method.
• Ethical considerations may preclude the use of
  experiments on human subjects.
• Biases have to be carefully addressed with this
  methodology.
• Costs of control and randomization are a
  consideration.

11
An Example of the Experimental Methodology

• Wildemuth, Barbara M., Charles P. Friedman, and
  Stephen M. Downs. Hypertext versus Boolean
  Access to Biomedical Information A Comparison
  of Effectiveness, Efficiency, and User
  Preferences. ACM Transactions on Computer-Human
  Interaction. 5.2 (1998) 156-183.

12
Research Questions

• To what extent does information retrieved from a
  database help students solve problems in
  bacteriology?
• Does this effect vary as students levels of
  personal domain knowledge increase?
• Is Boolean or hypertext access to a database more
  effective in supporting problem solving?
• Is one of these database access mechanisms more
  efficient than the other, in terms of its effect
  on problem-solving performance?
• Do students prefer one access mechanism over the
  other?

13
Experimental Method and Our Article

• Randomization Simple
• Sample size 82
• Control exerted through independent
  variables 1) access mode to which student is
  assigned 2) problem set order 3) assessment
  occasion
• Hypothesis (1) database use is an effective
  supplement to personal knowledge in a
  problem-solving context and (2) the level of the
  searchers personal knowledge in the relevant
  domain will affect the number and variety of
  search terms selected

14
Methodology and Our Article

• Randomization?
• Pretest and Posttest
• Statistical tests
• ANOVA
• Chi-square
• T-test

15
Pretest/Posttest Procedure 1994 1995 1st year
Medical Students

• Pretest given in November before class in
  microbiology.
• Without database own knowledge
• With database
• Posttest given in March after class in
  microbiology
• Without database own knowledge
• With database

16
Mean test scores
17
ANOVA Testing of Assessment (mean scores)
18
Total Improvement Mean Scores
19
1995 Students
November test time averaged 96 minutes, March
test time averaged 50 minutes.
20
Chi-square Testing

• The researchers used chi-square testing to
  determine if the students preferred hypertext or
  Boolean access.
• The researchers conclude there was no significant
  difference in effectiveness or efficiency between
  hypertext and Boolean.
• Chi-square results indicated students preferred
  Boolean access over hypertext.

21
Second Look Effect

• Students can take extra time to look at
  questions.
• Jog memory.
• Operates equally in both groups.
• Should not constitute any additional bias.

22
Discussion of Results

• The interaction between system preference and
  system assigned did not have a statistically
  significant effect on performance.
• Students with high or low personal knowledge can
  retrieve relevant information from a database.
• Both access methods were deemed equally helpful
  and used with equal efficiency.
• Hypertext might encourage persistence Boolean
  user might conclude prematurely that no useful
  information exists.