Thumbnail sketch of current experimental projects

1
Thumbnail sketch of current experimental projects

• Voter turnout and persuasion
• Downstream experiments habit, education and
  political participation, lottery winners
• Judges, sentencing, and recidivism
• Outward Bound and prejudice reduction
• Teaching strategic reasoning
• Crack house raids and deterrence
• Detecting cheating on college exams

2
Why Experiment?

• Causal explanation vs. description
• Parameters and parameter estimation
• Explanation, counterfactuals, and the role of
  randomization

3
The scope of causal questions

• Program evaluation SAT prep classes, weight loss
  programs, foster homes, fundraising, diversity
  training, deliberative polls, virginity pledging,
  advertising campaigns, wilderness programs
• Public policy evaluation speed traps, vouchers,
  alternative sentencing, job training, health
  insurance subsidies, tax compliance, public
  housing
• Behavioral Research persuasion, mobilization,
  education, income, interpersonal influence,
  conscientious health behaviors
• Research on Institutions rules for authorizing
  decisions, rules of succession, manner in which
  an organization is founded, distribution of power

4
What is an experiment?

• Experiment in common parlance vs. experiment as
  a term of art random assignment of observations
  to treatment and control conditions
• Perfectly controlled experiments, randomized
  experiments, and quasi-experiments
• Continuum of exogeneity generated by
  randomization procedures? Or hand-waving?
• Confusion between random sampling and random
  assignment definitions

5
Are experiments feasible?

• Lost history of social science experiments field
  vs. laboratory
• Working with the powerful but indifferent
• The persuasive power of randomized
  experimentation
• Seizing opportunities presented by naturally
  occurring randomized assignment

6
Consider the alternatives to experiments

• Survey analysis sampling advantages, but problem
  of measurement and unobserved heterogeneity
• Process tracing useful for developing hypotheses
  but often subject to uncertain inference due to
  limited sample size and unobserved heterogeneity
• Econometric analysis cross-sections,
  time-series, and panels are all subject to
  uncertainty about measurement and unobserved
  heterogeneity
• Dilemma study the big questions or the tractable
  questions?

7
Selecting the research question

• Confirmatory vs. exploratory investigation
• Advantage of experimentation Making one think
  clearly about hypotheses before the empirics are
  launched (note the tradeoff between creativity
  and excessive discretion in scientific
  interpretation)
• What is the independent variable? What is the
  dependent variable?
• What are the practical or theoretical
  implications of estimating this causal
  relationship?

8
Determining the unit of analysis

• Individuals, groups, institutionsor many
  potential levels
• Choice of analytic level determines what the
  study has the capacity to demonstrate.
• Example randomize school vouchers at the level
  of the individual or at the level of the
  community? Do we want to know how students
  respond to changing environment or how schools
  respond to competition?
• What kind of control group? Importance of
  placebo control groups in medicine

9
Integrity of Randomization

• Strict definition of random importance of
  procedures for generating random patterns
• Randomness creates predictable statistical
  patterns, but such patterns cannot guarantee that
  the data were generated randomly (or vice versa)
• Random assignment can be undone by human mischief
  (reassignment, multiple applications, tampering,
  administrative errors or sample attrition that
  disproportionately affects one group)

10
Randomization Pre-stratification?

• Distinction between Simple and Matched (or
  pre-stratified) Randomization
• Example four subjects with pre-test scores of
  2,2,8,8, divided evenly into treatment and
  control groups, and treatment effect is zero
  simple random assignment will place 2,2 and
  8,8 together in the same treatment or control
  group 1/3 of the time. Pre-stratification will
  ensure that the groups start out with identical
  characteristics.

11
Pre- vs. Post-stratification

• Concept of post-stratification in the absence
  of pre-stratification, one can still control
  for pre-treatment differences. In this case,
  subtract post-treatment scores from pre-treatment
  scores. In the general case, use multivariate
  regression, with pre-test scores as control
  variables.
• Pre-stratification vs. Post-stratification
  without covariates, pre-stratification can be
  much better with covariates, slight differences
  in efficiency, most noticeable in small samples
• Pre-stratification can keep critics off your
  back, but it can make the design more complex and
  less transparent
• post-stratification has a faint odor of
  data-dredging, so its important to document ex
  ante what your statistical analysis will be!

12
If you stratify

• Dont forget to control for the units within
  which you randomized! You effectively have a
  separate experiment within each stratum. Here,
  weve randomized within 2 strata.

13
Doh!

• Notice what happens when you neglect to control
  for the strata! Statistically significant and
  misleading results

14
Failure-to-treat A fixable problem, unless you
make it worse

• People you fail to treat are NOT part of the
  control group!
• Base your estimators on the ORIGINAL treatment
  and control groups, which were randomly assigned

15
Failure-to-treat A fixable problem, unless you
make it worse

• People you fail to treat are NOT part of the
  control group!
• Base your estimators on the ORIGINAL treatment
  and control groups, which were randomly assigned

16
Example of a biased comparison

• What if we had compared those contacted and those
  not contacted in the phone bank study?

17
How to deal with failure-to-treat an example
from voter mobilization research

• Suppose that the world is divided into two kinds
  of people reachable and nonreachable
• Let Pr the probability that reachable people
  vote
• Let Pnr the probability that nonreachable
  people vote
• Let a the proportion of reachable people in the
  population
• Let T the effect of the treatment

18
Model voting rates in control and treatment groups

• Probability of voting in the control group is a
  weighted average of reachable and nonreachable
  voting rates
• Vc a Pr (1- a) Pnr
• Probability of voting among reachable people in
  the treatment group is their base rate plus the
  treatment effect
• Vr Pr T
• Probability of voting among nonreachable people
• Vnr Pnr
• Therefore Vt a (Pr T) (1- a) Pnr

19
Derive an Estimator for the Actual Treatment
Effect (T)

• Vt-Vc a(PrT)(1- a)Pnr a Pr (1- a) Pnr
• aT
• aT is the estimated intent to treat effect
• To estimate T, insert sample values into the
  formula
• T (Vt - Vc)/a
• where a is the proportion of reachable people
  observed in the treatment group and Vt and Vc are
  the observed voting rates

20
Example Door-to-door canvassing in New Haven,
1998 a .282
21
Example Vt.481, Vc.444
22
Estimate Actual Treatment Effect

• Replacing ( Vt Vc ) / a with sample values
• ( 48.1 44.4 ) / .282 13.1
• In other words actual contact with canvassers
  increased turnout by 13.1 percentage-points
• Notice that we NEVER compare the voting rates of
  those who were contacted to those who were not
  contacted!Why not?

23
Interpreting Regression Results Estimate
Intent-to-Treat Effect

• Dependent variable.. VOTE98 Listwise Deletion of
  Missing Data
• R Square .00087
• Adjusted R Square .00079
• Standard Error .49747
• Analysis of Variance
• DF Sum of Squares Mean
  Square
• Regression 1 2.8266
  2.8265739
• Residuals 13191 3264.5220
  .2474810
• F 11.42138 Signif F .0007
• ------------------ Variables in the Equation
  ------------------
• Variable B SE B Beta
  T Sig T
• PERSNGRP .036727 .010868 .029413
  3.380 .0007
• (Constant) .444329 .004836
  91.871 .0000

24
How to estimate actual treatment effects and
standard errors 2SLS

• Idea behind instrumental variables find an
  instrumental variable (Z) that is correlated with
  the independent variable (X) but uncorrelated
  with omitted causes of Y
• In practice Y is the dependent variable, Actual
  Contact is the independent variable, Random
  Assignment is the instrumental variable

25
Interpreting Regression Results Estimate Actual
Treatment Effect (2SLS)

• Dependent variable.. VOTE98 Listwise Deletion of
  Missing Data
• R Square .00087
• Adjusted R Square .00079
• Standard Error .49659
• Analysis of Variance
• DF Sum of Squares Mean
  Square
• Regression 1 2.8266
  2.8265739
• Residuals 13191 3252.9359
  .2466027
• F 11.46206 Signif F .0007
• ------------------ Variables in the Equation
  ------------------
• Variable B SE B Beta
  T Sig T
• CONTACT .130039 .038410 .060049
  3.386 .0007
• (Constant) .444329 .004828
  92.034 .0000

26
Summary Intent-to-treat vs. Actual treatment

• Failure to treat as a correctable problem when
  treatment effects are constant
• Are treatment effects constant across all
  subjects? If not, treatment effects are more
  accurately described as treatment effects among
  the treated.
• Does the treatment have the same effect on the
  intended treatment group as it does on the
  intended control group? (Example restorative
  justice)

27
Designing Experiments with Power in Mind

• Power probability of rejecting the null
  hypothesis (usually that there is no effect)
  given a treatment effect of a certain size and
  the characteristics of a proposed experiment
• Relevant characteristics sample size, fraction
  assigned to the treatment group, contact rates,
  disturbance variance
• See http//research.yale.edu/vote/stat103/power_ca
  lculations.xls

28
What if Your Experiment Has Low Power?

• Still may be worth doing! May contribute to the
  accumulation of knowledge
• May help rule out very large (or small) treatment
  effects
• But results should be interpreted with caution
  when low-power studies reject the nulljust a bit
  too lucky?

29
Downstream Experimentation

• Usefulness of truly exogenous change
• Example Habit and Voter Turnout
• Opportunities for multidisciplinary cooperation
  looking at a wide array of consequences flowing
  from a single causative agent
• Resuscitates laboratory experiments?
• Limitations only works for genuine experimental
  effects (not those generated by publication
  bias) will be powerful only when the
  experimental effects are large or obtained from
  large samples

30
Empirical Illustration of Downstream Analysis

• Control Treatment (Treatment - Control)
  Downstream
• Percent who
• Voted in 1996 54.65 54.54 -0.11
• Percent who
• Voted in 1998 (Y) 47.62 49.41 1.79
• Percent who
• Voted in 1999 (V) 38.81 39.77
  0.96 0.96/1.790.536
• N of cases 10,073 15,127
• Source Gerber, Green, and Shachar (2002). The
  treatment group consists of those who were
  contacted by mail or in person prior to the 1998
  election. Voting rates in 1996, prior to the
  randomized intervention, are shown here to
  illustrate the comparability of the treatment and
  control groups.

31
Practical Challenges of Implementing an
Experiment

• Your motivation academic rewards, practical
  rewards, costs vis-à-vis other types of research
  opportunities
• Motivating those who are contemplating and
  managing the intervention perceptions of outside
  meddling, perceived costs of subtracting control
  groups
• Motivating funders basic research vs. applied
  research program evaluation?
• Motivating participants to what extent are
  unobtrusive designs feasible?

32
Bureaucratic Considerations

• Human subjects committees confidentiality,
  intrusiveness, use of public records, special
  considerations for vulnerable populations, survey
  research exemptions
• Collaborating with organizations need for
  authoritative agreement from leaders
• Assembling a research team fine points of
  co-authorship
• Securing funding

33
Experimental Procedures

• Within subjects or between subjects?
• Post-test only? Or Pre-post?
• Importance of supervising randomization checking
  that randomization worked as expected (e.g.,
  mobilization in one election should be unrelated
  to turnout in a previous election)
• Researchers much assess the validity of the
  treatments contact rates
• Must measure the background or context in which
  the experiment occurs

34
Ethics and Ethos of Experimentation

• Ethics of intervention What kinds are justified
  and on what grounds? Who bears the costs/risks
  of adverse effects?
• Procedural responsibilities Informed consent?
  Conflicts of interest. University imprimatur and
  institutional review boards
• Campbells image of an experimenting society

35
Putting it down on paper Elements of a research
report

• Abstract give key information about hypothesis,
  sample, findings, and implications
• Introduction motivate and situate the current
  research program
• Thesis paragraph laying out structure of essay
• Methods section explain how randomization was
  conducted, manipulation check, measures used
• Model section describe the parameter(s) of
  interest and how it/they will be estimated, what
  tests will be used

36
Putting it down on paper, continued

• Research findings simpler is better, but control
  for all of the design effects use covariates
  (according to ex ante plan) to improve efficiency
• Draw statistical and substantive conclusions
• Indicate relevance of findings for ongoing
  research literatures
• Suggest fruitful lines for further inquiry