A Second Look at Research in Naturalistic Settings

1
A Second Look at Researchin Naturalistic Settings

• Graziano and Raulin
• Research Methods Chapter 13

2
Why Do Field Research?

• Many reasons for doing field research
• Test external validity of laboratory studies
• Do the laboratory findings really indicate what
  happens in the natural world?
• Study events that only occur in the field
• Evaluate the effectiveness of social programs
• Improve generalization across settings
• By testing your hypotheses in those settings
• Field research is becoming more common

3
Difficulties in Field Research

• More difficult to control variables in field
  studies
• Sometimes we have no control over the independent
  variable (e.g., a natural disaster)
• Often we cannot randomly assign our participants
  to conditions
• Even though we may not have complete control, we
  can often get sufficient control to allow us to
  draw valid conclusions

4
Quasi-Experimental Designs

• Similar to, but not quite equal to, an
  experimental design
• Approximates the control of an experiment
• Researcher does not control key variables
• Manipulation of the independent variable
• Assignment of participants to conditions
• Two types of quasi-experimental designs
• Nonequivalent control-group design
• Interrupted time-series design

5
Nonequivalent Control-Group Design

• Like the pretest-posttest, natural control-group
  design discussed in Chapter 10
• The more similar the groups are, the closer this
  design mimics an experiment
• Sometimes participants are, for all practical
  purposes, randomly assigned by nature
• Other times, the groups are clearly different
• Careful analysis and cautious interpretation may
  still produce valid conclusions

6
Interrupted Time-Series Design

• A single group of participants is tested
  repeatedly both before and after a manipulation
  or a natural event
• Essentially a single-group, pretest-posttest
  design with multiple before and after measures
• The multiple measures allow us to detect many
  confounding variables
• Regression to the mean, history, maturation, etc.

7
Sample Data
8
Single-Subject Designs

• Effort to create maximum experimental control in
  a single-subject situation
• Unlike case studies, the experimenter actively
  controls variables in these single-subject
  designs
• Skinner was a proponent of these designs
• Now widely used in behavior modification studies
• Some single-subject designs
• Reversal (or ABA) design
• Multiple baseline design
• Single-subject, randomized, time-series design

9
Reversal (ABA) Designs

• Control is demonstrated in the reversal design if
  the behavior changes at each reversal of
  conditions
• Useful only for independent variables with
  momentary effects
• Longer lasting effects would not reverse
• Often used in behavior therapy to demonstrate the
  source of reinforcement

10
Multiple Baseline Designs

• Treating several conditions, one after the other,
  with a similar treatment approach
• If each condition changes only when the treatment
  is applied to it, then it is likely that the
  treatment caused the observed changes
• Useful when
• Treatment produces a long-lasting effect
• The researcher does not want to withdraw
  treatment once things have improved

11
Single-Subject, RandomizedTime-Series Design

• Like the interrupted time-series design except
• There is only a single participant
• The timing of the manipulation is random
• If the dependent variable changes at the precise
  time of the independent variable manipulation, it
  is unlikely to be due to anything other than the
  manipulation

12
Program Evaluation

• One of the fastest growing areas of research
• Demand to evaluate the effectiveness of costly
  programs before they are renewed
• Program evaluation can provide field tests of
  laboratory theories
• Program evaluation uses the designs and controls
  already covered, with a few twists
• Making the designs work in the field is a
  challenge

13
Practical Problems in Evaluations

• In program evaluation, the participants have
  not volunteered for research
• Instead, they are a part of the program
• Raises ethical issues of consent
• Including a no-treatment control group may raise
  both ethical and political questions
• Treatment staff may be threatened by the
  evaluation or irritated by the extra workload

14
Control in Program Evaluation

• Selecting appropriate dependent measures
• Normally need several dependent measures to
  evaluate all of the factors expected to change
• Should include at least one satisfaction measure
• Avoiding bias in the evaluation
• Use objective measures whenever possible
• Measure community-wide effects, which are less
  susceptible to response set biases
• Use strongest research design possible

15
Program Evaluation Designs

• Randomized control-group design
• The ideal excellent control, but may be
  impossible
• Nonequivalent control-group design
• Natural control groups may add much to an
  otherwise weak design
• Single-group time-series design
• Best alternative if no control group is available
• Pretest-posttest design
• A weak design not recommended

16
Summary

• Field research is becoming more common
• Field research need not be low constraint
• Types of field research
• Quasi-experimental designs
• Single-subject designs
• Program evaluation
• Field studies ultimate test of a hypothesis
• Is this how nature really works?