Lecture 19: Program Evaluation and Quasi-Experiments - PowerPoint PPT Presentation

1 / 23
About This Presentation
Title:

Lecture 19: Program Evaluation and Quasi-Experiments

Description:

Lecture 19: Program Evaluation and Quasi-Experiments Correlation and Causality Fact 1: Correlation does not imply causality. Fact 2: Causality implies correlation. – PowerPoint PPT presentation

Number of Views:111
Avg rating:3.0/5.0
Slides: 24
Provided by: msuEduco84
Learn more at: https://www.msu.edu
Category:

less

Transcript and Presenter's Notes

Title: Lecture 19: Program Evaluation and Quasi-Experiments


1
Lecture 19 Program Evaluation and
Quasi-Experiments
2
Correlation and Causality
  • Fact 1 Correlation does not imply causality.
  • Fact 2 Causality implies correlation.
  • A correlational study can provide evidence
    AGAINST a causal hypothesis. The problem is that
    a correlational study (by itself) does not give
    us much confidence when making affirmative causal
    inferences.

3
Program Evaluation
4
What is Program Evaluation?
  • The use of social science methods to
    systematically investigate the effectiveness of
    social intervention programs.
  • Focus on summative evaluation. Formative
    evaluation is where the goal is to help improve
    existing programs rather than making
    effectiveness judgments.
  • Senator D. P. Moynihan (1927-2003) If there is
    any empirical law that is emerging from the past
    decade of widespread evaluation activity, it is
    that the expected value for any measured effect
    of a social program is zero.

5
Reforms as Experiments (Campbell, 1969)
  • Problem Specific programs are often
    advocated as though they were certain to be
    successful.
  • Campbells Solution Hard-Headed Evaluation of
    Effects. Focus on the Importance of the Problem.
    Demand Empirical Evidence that Programs Work.
  • Political Stance This is a serious problem. We
    propose to initiate Policy A on a trial basis.
    If after five years there has been no significant
    improvement, we will shift to Policy B (p. 410)

6
Surefire Paths to Success (p. 428)
  • What to do if you want to see that your program
    works? Some ideasrely on testimonials and
    capitalize on regression artifacts
  • Human courtesy and gratitude being what it is,
    the most dependable means of assuring a favorable
    evaluation is to use voluntary testimonials for
    those who have had the treatment (p. 426)

7
Regression Toward the Mean
  • Extreme Scores at one time are not likely to be
    as extreme on a second testing.
  • Why? Two sets of scores are never perfectly
    correlated.
  • Take those 25 people who scored 65 or worse on
    Exam 1. What was their average gain from Exam 1
    to Exam 2? 4.10 points. What about those 13
    people who scored 95 or better? What was their
    average difference? A loss of 2.34 points.

8
Regression to the Mean - 2
  • This true story illustrates a saddening aspect
    of the human condition. We normally reinforce
    others when their behavior is good and punish
    them when their behavior is bad. By regression
    alone, therefore, they are most likely to improve
    after being punished and most likely to
    deteriorate after being rewarded. Consequently,
    we are exposed to a lifetime schedule in which we
    are most often rewarded for punishing others, and
    punished for rewarding. (Kahneman Tversky,
    1973, p. 251).

9
How could you use this phenomenon to create a
positive impression of a program?
10
Death, Taxes, and Regression to the Mean
  • Regression to the mean is as inevitable as death
    and taxes. Academic performance, emotional
    well-being, medical diagnosis, investment return,
    athletic feats, motion picture sales, and any
    other variable you can think of all exhibit
    regression toward the mean. (Reichardt, 1999, p.
    ix)

11
Applied Research
12
Example
  • Palmgreen, P., Donohew, L., Lorch, E. P., Hoyle,
    R. H., Stephenson, M. T. (2001). Television
    campaigns and adolescent marijuana use Tests of
    sensation seeking targeting. American Journal of
    Public Health, 91, 292-296.

13
Palmgreen et al. (2001)
  • Objective To evaluate the effectiveness of one
    television media campaign designed to reduce
    marijuana use among an at-risk group, high
    sensation seekers.
  • Sensation seeking is a trait associated with the
    need for novel and intense stimuli and the
    willingness to take risks to obtain such stimuli.
  • Design 32-month interrupted time series design.
  • Logic Before and After Comparisons. Cross-site
    comparisons.

14
Data Collection Basics
  • Duration Beginning 8 months before the first
    campaign and lasting 8 months after the last
    campaign.
  • Sites Two Counties in Southern States.
  • Each month draw a random sample of 100 public
    school students in each county
  • Measures Sensation Seeking and 30-day use of
    ATOD.

15
(No Transcript)
16
Interrupted Time Series Design
  • Extension of the pretest-posttest design
  • A stronger argument can be made to eliminate
    maturation, testing, and history effects
  • Can also be used with multiple groups with and
    without the treatment

Group 1
O2
O3
O1
O5
O6
O4
X1
17
Replicated Interrupted Time Series Design 2 (p.
323)
Group 1
O2
O3
O1
O5
O6
O4
X1
Group 2
O8
O9
O7
X2
O13
O11
O10
  • Both groups are exposed to treatment but at
    different times.
  • Quite strong from the point of view of internal
    validity
  • Why?

18
What makes this a quasi-experimental design?
  • One or more independent variables are being
    manipulated but participants are not randomly
    assigned to conditions.
  • What is the textbook threat in this case?
  • How plausible is this threat?
  • Quasi-experiments, however, rely heavily on
    researcher judgments about assumptions,
    especially on the fuzzy but indispensable concept
    of plausibility. (Shadish et al., 2002, p. 484)

19
Quasi-Experimental Designs
20
Notation
  • X IV, a treatment, or putative (supposed)
    cause
  • O DV, an observation, or putative effect
  • Notation used in Campbell and Stanley (1966)
  • Recall Threats Selection, Maturation, History,
    Instrumentation, Attrition

21
One-Shot Case Study
  • Single Group Studied Once
  • As been pointed out (e.g., Boring, 1954
    Stouffer, 1949) such studies have such a total
    absence of control as to be of almost no
    scientific value (Campbell Stanley, 1966, p.
    6). Basic to scientific evidence is the process
    of comparison. There is no point of comparison
    here. Misplaced precision.

X
O1
22
Static-Group Comparisons
  • The dashed line indicates a lack of random
    assignment
  • Selection is a serious threat to internal
    validity
  • Temporal precedence is often hard to establish

Group 1
X
O1
Group 1
X1
O1
OR
Group 2
X2
O2
Group 2
Not X
O2
Group 3
X3
O3
23
Pretest-Posttest Nonequivalent control group
design
  • Can help evaluate the extent to which selection
    is a threat to validity
  • Temporal precedence is clear

Group 1
X1
O2
O1
Group 1
X1
O2
O1
OR
Group 2
X2
Group 2
O4
O3
O4
O3
Group 3
X3
O6
O5
Write a Comment
User Comments (0)
About PowerShow.com