Experimental Research Validity and Control

1
Experimental ResearchValidity and Control
2
Visualizing Designs

• Structure of a research project
• Shows at a glance the major components of an
  experiment
• Design notation
• Useful for seeing the temporal sequence of the
  experiment
• Block notation
• Useful for visualizing conditions, interactions

3
Design Notation

• Elements
• Observations/Measures O
• Treatments/Programs X
• Groups Each group has its own line
• Assignment to group
• R Random assignment
• N Nonequivalent groups
• C Assignment by cutoff
• Time Moves from left to right

4
Design Notation

• Pretest-posttest design, two groups

R O X O R O O
5
Design Notation
R O1 X O1,2 R O1 O1,2
6
Classifying designs
R X O R O
Posttest-only randomized experiment
N O X O N O O
Pre-post nonequivalent groups quasi-experiment
Posttest-only nonexperiment
X O
7
Block Notation

• Each axis contains one IV

8
Block Notation

• Each block contains one condition

Independent Variable Type of Noise
Experimental group 2
Experimental group 3
Experimental group 1
White noise
Music
No noise
9
Music study from slide 7 in design notation
R X1 O R X2 O R O
10
Establishing Cause and Effect

• Temporal Precedence
• Cause has to happen before the effect
• Covariation of Cause and Effect
• Show there is a relationship between X and Y
• No Plausible Alternative Explanations
• Any other cause that can bring about the effect

11
Establishing Cause and Effect
12
Validity

• Measure-Related Concerns
• Construct Validity
• Are you measuring what you think youre
  measuring?
• Reliability
• Is your measure consistent in its evaluation of
  the same individuals?
• Conclusion Validity
• Are the conclusions you reach about the
  relationships in your data reasonable?

13
Validity

• Measure-Related Concerns
• Construct Validity
• Reliability
• Conclusion Validity
• Method-Related Concerns
• Internal Validity
• External Validity

14
Validity

• Internal Validity
• Is the independent variable the only possible
  explanation of the results shown?
• Protecting internal validity allows us to
  eliminate potential alternative explanations for
  the outcome of an experiment
• Extraneous variables
• Influence the DV without your intent or consent!

15
Experimental Control

• Extraneous Variables
• Nuisance variables
• Make the effects of the IV more difficult to
  determine by affecting variation within groups
• Participant characteristics
• Unintended influences of experimental situation

16
Experimental Control

• Extraneous Variables
• Confounding variables
• Vary systematically with the IV
• Influence the difference between groups
• The differences observed between groups could be
  attributed to the IV or the confound

17
Experimental Control

• Extraneous Variables
• Experimenters
• Physiological differences
• Rosenthal effects

18
Experimental Control

• Extraneous Variables
• Participants
• Demand characteristics
• Good participant effect
• Response biases

19
Internal Validity

• Exercise control to protect internal validity
  before the experiment is conducted
• Random Assignment
• Elimination
• Constancy
• Counterbalancing
• Single- and Double-blind Experiments
• Altering Response Sets

20
Basic Control Techniques

• Randomization
• Individuals have an equal chance of being
  assigned to any group in the experiment
• Allows us to assume groups are roughly equivalent
  before administering the IV

21
Basic Control Techniques

• Elimination
• Extraneous variables are completely removed from
  the experiment

22
Basic Control Techniques

• Constancy
• An extraneous variable is reduced to a single
  value experienced by all participants
• Standard control technique
• Hold environment, temperature, lighting constant

23
Basic Control Techniques

• Balancing and Counterbalancing
• Balancing
• Achieves group equality by distributing
  extraneous variables to all groups
• Counterbalancing
• Controls order effects by presenting treatments
  in a different sequence
• Order or sequence effects
• Carryover effects
• Differential carryover

24
Basic Control Techniques

• Within-Subject Counterbalancing
• Within-Group Counterbalancing
• Each treatment must be presented to each
  participant an equal number of times
• Each treatment must occur an equal number of
  times at each testing session
• Each treatment must precede and follow each of
  the other treatments an equal number of times

25
The Cola Wars!

• Participant 1
• Participant 2
• Participant 3
• Participant 4
• Participant 5
• Participant 6

• P C
• P C
• P C
• C P
• C P
• C P

26
The Cola Wars!

• Participant 1
• Participant 2
• Participant 3
• Participant 4
• Participant 5
• Participant 6

• P C DP
• P DP C
• C P DP
• C DP P
• DP P C
• DP C P

27
Basic Control Techniques

• Single- and Double-Blind Experiments
• The experimenter (single-blind) or both the
  experimenter and participants (double-blind) are
  unaware of the participants condition

28
Basic Control Techniques

• Altering response sets
• Rewrite survey items so that some negative
  responses indicate agreement and some positive
  responses indicate disagreement
• Check for socially desirable responses in your
  survey and/or potentially undesirable cues

29
Internal Validity

• Exercise control to protect internal validity
  before the experiment is conducted
• Assess the internal validity of the experiment
  afterward

30
Threats to Internal Validity

• History
• Significant events occur between measurements

X O
R O X O R O O
O X O
31
Threats to Internal Validity

• Maturation
• Participants change over time in the experiment

O X O
32
Threats to Internal Validity

• Testing (Pretest Sensitization)
• Repeated tests cause changes in DV practice
  effects

O X O
GRE
33
Threats to Internal Validity

• Instrumentation
• Changes in measurement due to changes in the
  measurement device (human or machine)

O X O
Paper GRE
Computer GRE
34
Threats to Internal Validity

• Statistical regression
• Extreme scores regress toward the mean
• Difficult to maintain extreme scores over
  repeated measures

35
Threats to Internal Validity

• Selection
• Selecting participants into groups that were not
  equivalent before the experiment

N X O N O
R X O R O
36
Threats to Internal Validity

• Attrition or Mortality
• Experimental dropouts

37
Threats to Internal Validity

• Diffusion of treatment
• Participants in one group communicate information
  from the IV to the other group

38
Threats to Internal Validity

• Interactions with Selection
• Maturation
• History
• Instrumentation

39
How Important is Internal Validity?

• It is the most important property of any
  experiment
• An experiment without internal validity cannot
  have external validity
• With no internal validity, you can have no
  confidence in your results

40
External Validity

• How far can you generalize your results beyond
  your experiment?
• 3 types
• Population
• Environmental
• Temporal

41
Threats to External Validity

• Testing/Treatment Interaction
• Pretest may make reaction to the IV different
  from those not tested
• Selection/Treatment Interaction
• Effect is found only for a specific group of
  participants

42
Threats to External Validity

• Reactive Arrangements
• A la Hawthorne effects
• Demand characteristics
• Participant Characteristics
• Specific animal species

43
(No Transcript)
44
Threats to External Validity

• Reactive Arrangements
• A la Hawthorne effects
• Demand characteristics
• Participant Characteristics
• Specific animal species
• Experimental participant populations
• College students
• White male Americans

Experimental setting alters participants behavior
45
Is External Validity Necessary?

• May want to find out if something can happen
• We may be predicting from the real world to the
  lab
• If we can show something happens in the labs
  unnatural setting, we may have more confidence in
  the phenomenon
• We may study something without a real-world
  analogy