Field Experiments in Political Science: A Brief History and Some Illustrative Examples

1
Field Experiments in Political Science A Brief
History and Some Illustrative Examples

• Don Green
• Columbia University

2
What is an experiment?

• Units of analysis are randomly assigned with
  known probability to treatment and control
  conditions
• Distinction between random and haphazard
  assignment
• Distinction between random assignment and random
  sampling
• In the physical sciences, perfectly controlled
  experiments substitute for randomized trials

3
What is a field experiment?

• Random assignment takes place in a naturalistic
  setting, enhancing generalizability
• Ideally, experimenters play an unobtrusive role,
  reducing the risk of a violation of symmetry
  between treatment and control
• Four dimensions of naturalism subjects,
  treatments, contexts, outcome measures

4
Early (near random?) field experiments in
Political Science

• Harold Gosnell Mobilizing voters in the 1924 and
  1925 elections
• George Hartmann Using rational or emotional
  appeals to increase the socialist vote in
  Allentown, PA in the 1935 elections
• Underhill Moore and Charles Callahan examined
  the effects of varying New Havens parking
  regulations, traffic controls, and police
  enforcement in 1930s

5
Early Field Experiments using Random Assignment

• Hovland, Lumdsdaine, and Sheffield Propaganda
  studies conducted for Experimental Section of the
  Research Division of the War Department
• Samuel Eldersveld Partisan and nonpartisan voter
  mobilization campaigns in the 1953 and 1954 Ann
  Arbor, MI elections
• Ironic that randomized field experiments should
  die out just as they are gaining prominence in
  other disciplines (e.g., polio vaccine trials)

6
Dominant Modes of Behavioral Research in
Political Science

• Surveys spearheaded in the 1950s by American
  National Election Studies using random sampling
• Econometric analysis of aggregate data over time
  and/or space growing computing power and
  technical facility of the discipline during 1970s
• Fundamental trade off between research design and
  post hoc statistical correctives

7
Illusion of Observational Learning Theorem
(Gerber, Green, and Kaplan 2004)

• When confronted with mixture of observational and
  experimental evidence, Bayes Rule says assign
  zero weight to observational evidence unless you
  have informative priors about its bias
• When confronted with laboratory experimental
  evidence, assign it zero weight unless you have
  informative priors about the biases associated
  with your extrapolation to some
  population/setting of interest

8
Implications of the Illusion for scientific
practice

• Researchers using observational data are
  oblivious to the fact that they routinely
  underreport the true degree of statistical
  uncertainty associated with their findings
• Misallocation of the disciplines research
  portfolio No field experiments conducted between
  1985 and 1998
• Misconceptions about meta-analysis

9
Reclaiming the Experimental Tradition in Social
Science

• Reassessment of evidence for a variety of basic
  behavioral propositions by subjecting them to the
  same level of scrutiny as pharmaceutical
  evaluations
• Secondary aim is to stimulate experimental
  reflection even in domains of political science
  where experimentation is infeasible
• What follows is a brief overview of selected
  projects, some of which refute the it cant be
  done critique

10
Project 1 Voter Mobilization(Authors Gerber
and Green)

• 1998 New Haven Study, N31,098
• Interventions nonpartisan face-to-face
  canvassing, commercial phone banks, direct mail
• Voter turnout measured using public records
• Subsequently replicated with hundreds of
  thousands of observations in a variety of sites

11
Synthesis of recent randomized experiments on
voter turnout door-to-door canvassing,
leafleting, phone calls, direct mail, and e-mail
12
Forest Plot of 85 direct mail experiments
excluding social pressureStudiesATE 0.109
ppts(-0.07,0.290)
13
Lessons Learned

• Face-to-Face canvassing raises turnout by
  approximately 7-9 percentage-points
• Volunteer phone banks have moderate effects (3-5
  percentage-points)
• commercial phone banks are typically ineffective,
  as are robo-calls
• Direct mail has weak effects (except as noted
  below)
• E-mail has no apparent effect

14
Project 2 Habit Formation(Authors Gerber,
Green, and Shachar)

• Infer habit from long-term effects of randomized
  intervention
• Example of downstream experimentation
• Follow up study of people in the 1998 New Haven
  Study showed that treatment group voted at higher
  rates
• Subsequently replicated in 4 of 5 studies
• For each 100 additional votes generated in this
  election, an additional 33 votes are generated in
  the next election

15
Project 3 Interpersonal Influence(Author
Nickerson)

• Inference without problems of unobserved
  heterogeneity, which plague other influence
  studies
• Placebo-control design 2 voter households
  receive either get-out-the-vote message or a
  recycling appeal
• Also, a control group gets nothing (as expected,
  turnout is significantly higher in GOTV vs.
  control)

16
Project 3 Interpersonal Influence(Author
Nickerson)

• Findings show that housemates of registered
  voters who were contacted in the treatment group
  were significantly more likely to vote than
  housemates of those who were contacted in the
  placebo group

17
Example Nickersons influence experiment in
Denver 2002
18
Project(s) 4 Influence of Television and Radio

• Cable system experiments in 2003, 2004 influence
  of GOTV ads on turnout
• Radio ads in 2005 and 2006 municipal elections
  influence on competitiveness
• Broadcast TV, cable TV, and radio in the context
  of a 2 million gubernatorial campaign
• Influence of radio on ethnic reconciliation in
  Rwanda
• Spanish language radio and voter turnout

19
Project 5 Social Pressure and Voter Turnout
(Gerber, Green, and Larimer)

• Using field experiments to test basic behavioral
  theories
• Longstanding interest in prescriptive norms
  dating back to Gosnells work in the 1920s
• To what extent can one manipulate the salience of
  extrinsic incentives associated with voting?

20
Study Design August 2006

• Sample 180,002 households in Michigan,
  registered voters who voted in 2004
• Setting August primary election, open but
  contested only on the Republican side
• Assignment 10,000 clusters of 18 households
  each in each cluster, households assigned at
  random to one of five groups Control, Civic
  Duty, Hawthorne, Self, and Neighbors
• Outcome Voting in the primary election, as
  indicated by official records for each individual

21
For more information [phone] email
etov_at_grebner.com Practical Political
Consulting P. O. Box 6249 East Lansing, MI
48826 ECRLOT C050 THE WAYNE FAMILY 9999 OAK
ST FLINT MI 48507 Dear Registered Voter WHO
VOTES IS PUBLIC INFORMATION! Why do so many
people fail to vote? We've been talking about
the problem for years, but it only seems to get
worse. This year, we're taking a different
approach. We are reminding people that who votes
is a matter of public record. The chart shows
your name from the list of registered voters,
showing past votes, as well as an empty box which
we will fill in to show whether you vote in the
August 8 primary election. We intend to mail you
an updated chart when we have that
information. We will leave the box blank if you
do not vote. DO YOUR CIVIC DUTY -
VOTE! --------------------------------------------
--------------- OAK ST
Aug 04 Nov 04 Aug 06 9999 ROBERT SMITH
Voted ______ 9999 LAURA BETH
Voted Voted ______
21
22
For more information [phone] email
etov_at_grebner.com Practical Political
Consulting P. O. Box 6249 East Lansing, MI
48826 ECRLOT C050 THE JACKSON FAMILY 9999
MAPLE DR FLINT MI 48507 Dear Registered
Voter WHAT IF YOUR NEIGHBORS KNEW WHETHER YOU
VOTED? Why do so many people fail to vote? We've
been talking about the problem for years, but it
only seems to get worse. This year, we're taking
a new approach. We're sending this mailing to you
and your neighbors to publicize who does and does
not vote. The chart shows the names of some of
your neighbors, showing which have voted in the
past. After the August 8 election, we intend to
mail an updated chart. You and your neighbors
will all know who voted and who did not. DO YOUR
CIVIC DUTY - VOTE! -------------------------------
---------------------------- MAPLE DR
Aug 04 Nov 04
Aug 06 9995 JOSEPH JAMES SMITH Voted
Voted ______ 9995 JENNIFER KAY SMITH
Voted ______ 9997
RICHARD B JACKSON Voted
______ 9999 KATHY MARIE JACKSON
Voted ______ 9999 BRIAN JOSEPH
JACKSON Voted ______ 9991
JENNIFER KAY THOMPSON Voted
______ 9991 BOB R THOMPSON
Voted ______ 9993 BILL S SMITH
______ 9989 WILLIAM
LUKE CASPER Voted
______ 9989 JENNIFER SUE CASPER
Voted ______ 9987 MARIA S JOHNSON
Voted Voted
______ 9987 TOM JACK JOHNSON
Voted Voted ______
22
23
2006 August Primary Election Experimental Group Experimental Group Experimental Group Experimental Group Experimental Group
Control Civic Duty Hawthorne Self Neighbors
Percent Voting 29.7 31.5 32.2 34.5 37.8
N of Individuals 191,243 38,218 38,204 38,218 38,201
All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level). All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level). All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level). All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level). All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level). All contrasts with the control group are significant at p lt .001, two-tailed test, using robust cluster standard errors (clustered at the household level).
24
Project 6 The Effects of Criminal Sentences on
Recidivism (Green and Winik)

• Random assignment of judges creates analytic
  leverage
• 1000 defendants in Washington, DC drug courts
  randomly assigned to nine calendars with
  different sentencing proclivities
• No effect of incarceration or length of sentence
  on recidivism

25
Miscellaneous Randomized Experiments of Interest
in the Domain of Political Attiudes and Actions

• Randomly varying rules about legislative
  seniority, term length, and floor recognition
• Discrimination experiments focusing on the
  responsiveness of legislators to constituents of
  varying ethnic, racial, or partisan profile
• Effects of audits and accountability
  interventions
• Exposure to draft, school, or visa lotteries and
  their effects on attitudes and behavior

26
Bottom Line

• Randomized experimentation in real world settings
  give social scientists access to the kinds of
  practical knowledge that outsiders care about
• Possibility, of course, for premature
  extrapolation of experimental results e.g.,
  class size experiments
• Importance of creating firm empirical foundation
  for theoretical development and policy
  intervention (e.g., prejudice reduction)

27
For a recent (2012) discussion of social science
experiments and experimental design, see