Researching Development: Strengths and Limitations of Field and Natural Experiments

1
Researching Development Strengths and
Limitations of Field and Natural Experiments

• Thad Dunning
• Department of Political Science
• Yale University

2
Randomized Controlled Experiments Published in
the American Political Science Review (1906-2004)
Source Jamie Druckman, Donald P. Green, James H.
Kuklinski, and Arthur Lupia. 2006. The Growth
and Development of Experimental Research
Political Science. American Political Science
Review 100 627-635.
3
Natural Experiments in Political Science and
Economics (1960-2005)
Articles that claim to exploit natural
experiments and that were published in political
science and economics journals, as tracked by
JSTOR
4
Outline

• Field and Natural Experiments The Methodological
  Rationale
• Substantive Examples in the Study of Development
• Ethnic Voting in Sub-Saharan Africa
• The Effects of Land Titling in South America
• The Salience of Caste in India
• Field and Natural Experiments as Design-Based
  Methods
• Strengths and Limitations of Field and Natural
  Experiments for the Study of Development

5
The problem of confounding

• Causal inference often depends on comparisons
  between groups of units (countries, individuals,
  etc.)
• Some units are exposed to a treatment,
  intervention, or independent variable and
  some are not
• Attributing differences between groups to the
  causal effect of the intervention or treatment
  can be misleading
• There may confounding factors -- factors
  associated with both the putative cause and the
  putative effect

6
Questions and confounders some examples

• How does the ethnic relationship between voters
  and politicians influence vote choice?
• Ethnicity may be related to confounding
  socio-economic characteristics that also shape
  vote choice
• Does establishing property rights for the poor
  raise incomes and reduce inequality?
• Access to land titling programs may depend on
  confounding individual characteristics
• How do institutions and political leadership
  affect the salience of different ethnic
  identities?
• Political institutions and leaders emerge from
  particular polities, and their character may be
  shaped by the salience of identities

7
The role of random assignment

• Random assignment to treatment ensures that, on
  average, potential confounders are balanced
  across the comparison groups
• As groups, the comparison sets are ex-ante
  identicalso, post-treatment differences between
  the groups can be more reliably attributed to the
  impact of the intervention
• Both field and natural experiments leverage the
  power of random (or as if random)
  assignmentbut these research designs differ in
  other fundamental ways

8
Randomized controlled experiments

• treatment and control groups
• random assignment
• manipulation

9
Natural experiments

• treatment and control groups
• random -- or as if random -- assignment
• no manipulation by the researcher

10
Natural Experiment An observational study
(that is, a study without an experimental
manipulation) in which assignment to treatment
and control groups is done at random -- or as
if at random
11
John Snows Natural Experiment

• Cholera in mid-nineteenth century London
• Against predominant theories, John Snow
  hypothesized that cholera was a water- or
  waste-born disease
• A natural experiment with which to evaluate the
  theory
• In 1852, the Lambeth water company moved its
  intake pipe upstream on the Thames, to a purer
  water source
• The Southwark and Vauxhall company left its
  intake pipe in place.

12
Death rate from cholera in London, by source of
water supply
(adapted from Snow 1855, Table IX)
13

• SNOW IN HIS OWN WORDS
• The pipes of each Company go down all the
  streetsA few houses are supplied by one Company
  and a few by the other, according to the decision
  of the owner or occupier at that time when the
  Water Companies were in active competition. In
  many cases a single house has a supply different
  from that on either side. Each company supplies
  both rich and poor, both large houses and small
  there is no difference either in the condition or
  occupation of the persons receiving the water of
  either company
• It is obvious no experiment could have been
  designed which would more thoroughly test the
  effect of water supply on the progress of cholera
  than this.
• -- John Snow (1885 74-75)

14
Field and Natural Experiments in the Study of
Development

• To what extent can research on development
  productively exploit field and natural
  experiments?
• Central challenges
• Key variables may not be manipulable by
  researchers (contra field experiments)
• In the observational world, treatment assignment
  is often highly non-random (contra natural
  experiments)
• Yet such approaches may play an important role in
  evaluating particular aspects of theories or in
  testing particular causal hypotheses

15
Example 1. A field experimentCross-cutting
cleavages and ethnic voting

• A puzzle despite Malis substantial ethnic
  heterogeneity, ethnicity plays a limited
  political role
• Parties do not form along ethnic lines
• Unlike many sub-Saharan African countries, in
  Mali ethnicity is a poor predictor of individual
  vote choice
• One hypothesis cousinage
• Malians with certain family names are linked
  through joking relations with their fictive
  cousins
• Two strangers -- say, with the patronyms Keita
  and Coulibaly, respectively -- may use cousinage
  relations to establish rapport and limit
  interpersonal conflict

16
Cousinage as a cross-cutting cleavage

• Cousinage relations occur within ethnic groups
  but also, especially, occur across ethnic groups
• Can such a cross-cutting cleavage inhibit ethnic
  conflict and undermine ethnically-dominated
  politics?
• Lots of theory and some observational evidence
• Dunning and Harrison (2008,2009) use an
  experimental design to estimate the effect of
  cousinage alliances and co-ethnicity on candidate
  preferences in Mali
• The experimental design can be applied in other
  contexts

17
Cousinage in brief

• Cousinage relations were codified during the rule
  of the emperor Sundiata Keita (c. 1235-1255) and
  exist in Mali, Sénégal, Guinea, the Gambia, the
  northern Ivory Coast, and western Burkina Faso
• Though various kinds of cousinage ties exist, we
  focus on cousinage alliances between patronyms
• Our focus is not on explaining the origins or
  persistence of cousinage alliances but rather on
  estimating their causal effects

18
An experiment in the field

• We showed videotaped political speeches with
  identical content to Malian participants
• Participants were told that the person in the
  video was considering running for National
  Assembly
• Experimental manipulation we varied the last
  name of the politician across different showings
  of the speech
• In Mali, last (family) name conveys information
  about both ethnic identity and cousinage
  relationships
• By varying the last name of the politician (as a
  function of the subjects last name), we hoped to
  manipulate the perceived cousinage and ethnic
  relationships of participants to the politicians

19
Experimental Design

• Subjects were assigned at random to one of six
  treatment and control conditions
• Treatment conditions
• Control conditions
• No last name given (N132)
• Subject and politician share patronym
  (N158)

Subject and politician are joking cousins
Subject and politician are not joking cousins
Subject and politician are from the same ethnic
group
Subject and politician are from different ethnic
groups
20
Subject recruitment

• A door-to-door canvass in all of Bamakos
  neighborhoods (quartiers), intended to be as
  representative of Bamako as possible
• After subjects agreed to participate, we obtained
  initial subject data, including last name and
  self-identified ethnicity
• Subjects were then randomized into one of six
  treatment conditions, shown the video, and
  administered a post-speech questionnaire

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Randomization of Treatment Assignment Creating a
cousinage map

• We used interviews with experts and ordinary
  Malians and drew on the literature to create a
  matrix
• the rows of which list potential last names of
  subjects
• the columns of which give last names associated
  with each of the treatment conditions
• This map of cousinage relations lists last
  names associated with each treatment condition,
  for more than 200 potential subject surnames
• We revised the matrix after initial field trials,
  using field interviews with informants as well as
  our initial data

25
A typical row of the random assignment matrix
(1) (2) (3) (4) (5) (6)
Co-ethnic/ cousin
Co-ethnic/ non-cousin
Non-coethnic/ cousin
Non-coethnic/ non-cousin
Same name
No name

• The final matrix has around 200 last names in
  the first column (the names for potential
  subjects)

26
Stimulating perceptions of co-ethnicity and
cousinage alliances

• Did participants infer their co-ethnic and
  cousinage relationships with the politician, as
  we intended? At the end of our survey, we asked
  subjects to identify the politicians ethnicity
  and whether he is a cousin.
• When given politicians last name, more than 70
  percent of subjects correctly identified
  politicians ethnicity (out of more than 11
  choices)
• When not given the politicians last name, a
  random guess based on the distribution of
  ethnicity in Bamako would have done just as well
• Participants also perceived the intended
  cousinage relationship to politicians with
  relatively high accuracy especially when
  presented with non-coethnics

27
(No Transcript)
28
(No Transcript)
29
Average candidate evaluations, by treatment
assignment
The figure displays average answers to the
question On a scale of 1 to 7, how much does
this speech make you want to vote for (name of
candidate)?
30
Other results (see paper!)

• The experimental data allows us to investigate
  why voters prefer their co-ethnics and their
  cousins
• Credibility (expectations about post-election
  behavior) appears to be more important than
  affective evaluations (likeability, etc.)
• We also test other implications of the argument
• For party strategy (parties appear to exploit
  cousinage networks in placing candidates on
  lists)
• For candidate behavior during campaigns
• Observational and experimental data are both
  useful in this regard
• Qualitative methods play a key role (e.g., in
  construction of the cousinage matrix)

31
Example 2. A natural experimentHow do
property rights affect the poor?

• A well-known hypothesis establishing property
  rights for the poor would raise incomes, increase
  investment, and allow access to capital (De Soto)
• Yet where property rights are extended, they may
  be extended to all similarly-situated people
• Or, their assignment may reflect confounding
  characteristics associated with particular poor
  citizens
• Galiani and Schargrodsky (2006, 2007) claim to
  exploit a natural experiment in Argentina, in
  which assignment to property rights is as if
  random

32
A natural experiment on property rights

• In 1981, squatters organized by the Catholic
  church occupied an urban wasteland in the
  province of Buenos Aires, dividing the land into
  similar parcels
• After the return to democracy, a 1984 law
  expropriated the land, with the intention of
  transferring title to the squatters
• Some of the original owners challenged the
  expropriation in court, leading to long delays,
  while other titles were ceded and transferred to
  squatters
• The legal action created a treatment group
  squatters to whom titles were ceded and a
  control group squatters whose titles were not
  ceded

33
A natural experiment on property rights

• Galiani and Schargrodsky (2006, 2007) argue that
  whether title was ceded is as if random
• One source of evidence is the pre-treatment
  equivalence of treated and untreated units
• Titled and untitled parcels are side-by-side
  (reminiscent of Snow)
• Pre-treatment characteristics of the parcels
  (such as distance from polluted creeks) are
  similar in both groups
• The compensation offered by the government (in
  square meter terms) was very similar across
  parcels
• Pre-treatment characteristics of squatters (age,
  sex, etc.) do not predict whether they received
  title
• Also, note that if the expropriation of only the
  more valuable parcels was challenged in court,
  this would imply that squatters in the control
  group had more valuable parcels
• Another important source of validation of the
  natural experiment is qualitative evidence for
  example, on the process by which the squatting
  took place

34
The effects of property rights

• Galiani and Schargrodsky (2006, 2007) find
  significant effects on housing investment,
  household structure, and educational attainment
  of children (but not on access to credit markets)
  
• They also find a positive effect of property
  right on beliefs in individual efficacy (!)
• The research design recalls John Snows natural
  experiment on cholera

35
Example 3. Combining Field and Natural
Experiments The Salience of Caste Categories

• Political leadership and the sanctioning of
  particular ethnic categories by the state may
  both shape the salience of different forms of
  ethnic identification
• In Indian villages, the election of lower-caste
  leaders is sometimes said to intensify conflict
  along caste-category lines (rather than, say,
  along sub-caste lines)
• Yet drawing causal inferences by comparing
  village councils with and without lower-caste
  presidents may be misleading
• My ongoing research in the state of Karnataka
  combines field and natural experiments to
  investigate the impact of reservation (an
  electoral quota for lower-caste politicians)

36
The Structure of Caste in Karnataka

• Two main jatis (sub-castes) comprise the
  Scheduled Castes
• The Holaya and the Madiga are former Untouchable
  sub-castes (Harijans), with some history of
  antagonism and competition
• There are two dominant jatis among the Backward
  Castes
• The Vokkaliga and Lingayath sub-castes tend to
  dominate politics at the local and state level
• So, what dimension of caste is most important to
  voters?
• E.g., Holaya/Madiga or Scheduled Caste?
• Vokkaliga/Lingayath or Backward Caste (anti-SC)?
• And do political institutions shape the relative
  salience of these categories?

37
Combining Field and Natural Experimental Evidence

• I implemented a field experiment in villages in
  the state of Karnataka, in which I manipulated
  the caste relationship between subjects and
  videotaped political candidates by changing the
  surname of candidates
• This experiment allows me to compare the
  influence on subjects preferences of
  relationships based on different caste
  categories?such as sub-caste (jati) or larger
  state-sanctioned groupings (e.g., Scheduled
  Castes)
• I embed this experiment in a special kind of
  natural experiment, based on a regression-discont
  inuity design
• I can then examine how reservation (required
  election of lower-caste leaders) shapes the
  salience of different caste categories

38
Design of the field experiment

• I filmed an actor delivering two versions of a
  political speech (in Kannada)
• During recruitment, subjects revealed their jati
  and several other attributes on a screening
  questionnaire
• Each subject was then assigned at random into one
  of the three treatment conditions
• Subject and politician have the same jati, same
  caste category
• Subject and politician have the same jati,
  different caste category
• Subject and politician have different jatis and
  caste categories
• Subjects were shown one of the two (identical)
  speeches, but field investigators varied the
  surname of the politician across different
  versions of the speech, according to treatment
  assignment

39
Experimental Design
Same caste category
Different caste category
Same sub-caste (jati)
Different sub-caste (jati)
40
Politicians surname by treatment condition
(Selected sub-castes)
Omitted subject sub-castes include Lambani
(SC), Kumbara (BC), and Bunt (BC)
41
Village Selection A regression-discontinuity
design

• I implemented the field experiment in 200
  villages in Karnataka, selecting participants at
  random within villages via stratified random
  sampling (stratified by caste)
• The key question is how to choose villages so
  that the causal effect of reservation can be more
  reliably estimated
• For this purpose, I used a regression-discontinui
  ty design, which is a special kind of natural
  experiment
• The general idea when assignment to a treatment
  variable (such as reservation) depends on the
  value of a covariate, and when there is a
  threshold that separates units assigned to
  treatment and those assigned to control, then
  units near the threshold are, on average, very
  similar
• Near the threshold, assignment to treatment may
  be as if random

42
Village Selection A regression-discontinuity
design

• Reservation for lower-caste politicians of the
  council presidencies is done as follows, starting
  with the 1994 elections
• In each taluk (an administrative unit below the
  district), a bureaucrat lists the Gram Panchayats
  in descending order, by the population proportion
  of each relevant group
• The proportion of presidencies that must be
  reserved for each group is determined by the
  groups population proportion in the taluk as a
  whole this also gives the number N of
  presidencies to be reserved
• Starting with the Scheduled Caste category, the
  bureaucrat moves down the list of Gram
  Panchayats, reserving the first N presidencies
• The same is then done for Scheduled Tribes the
  remaining Gram Panchayats are reserved for two
  groups of Backward Castes (A and B) or left for
  the General category
• Within each category, a third of the presidencies
  are also reserved for women
• In the next election, the bureaucrat takes up
  where he or she left off, rotating reservation to
  the next N villages on the list, for each
  respective category
• If, in any election, a Gram Panchayat is already
  reserved for one category (e.g., Scheduled Caste)
  but appears in among the GPs that should be
  reserved for andother category (e.g., Scheduled
  Tribe), the panchayat is skipped and then
  reserved for the latter category in the
  subsequent election
• There is also reservation of members seats,
  based on population proportions in each Gram
  Panchayat by ward (usually village)

43
The regression-discontinuity design (cont.)

• One can mimic the process of reservation, using
  census data on group proportions (the same data
  used by the bureaucrats)
• By listing panchayats in descending order of
  population proportion for each group, and using
  data on the reservation of presidencies for a
  given electoral term, one can find the threshold
  points that is, the cut-point between
  panchayats in each category that were reserved
  and those that were not
• The idea of the regression-discontinuity design
  is to select the villages on either side of, and
  nearest to, the threshold
• Suppose the cut-point is 18 in a given electoral
  cycle. Then whether a panchayat has 18.1
  Scheduled Castes and is thus reserved, or 17.9
  and is not reserved, is plausibly as-if random
• I did this for taluks in six districts of
  Karnataka and thereby built a population of 200
  panchayats, in which reservation is plausibly
  assigned as-if at random

44
Reservation as-if randomization checks
Mean male population here is omitted but also
passes the covariate balance test. P-value for
assignment covariates is 0.97 (SC proportion) and
0.26 (mean ST proportion).
45
Average candidate evaluations, by treatment
assignment

• On a scale of 1 to 7, how much does this speech
  make you want to vote for (name of candidate)?

The table gives averages. In the first three
rows, standard errors are in parentheses in the
final three rows, t-statistics are in
parentheses. Boldface type indicates that the
estimated effect is significant at standard
levels (plt0.05).
46
The causal effects of reservation
(1 same jati, same caste category 2
different jatis, same caste category 3
different caste categories)
47
The effects of reservation, in sum

• Reservation of the council presidency increases
  the salience of caste in shaping candidate
  evaluations
• Most importantly, reservation shifts the relative
  importance of caste categories, making the larger
  caste grouping relatively more important
• In general, reservation appears to most strongly
  shapes affective preferences (though this finding
  is preliminary)

48
Methodological reflections

• Field and natural experiments are both
  design-based rather than model-based methods,
  in that the most crucial decisions regard
  research design
• Attempts to confront inferential problems (such
  as confounding) come at the research design
  stage, rather than through ex-post statistical
  adjustment
• In other words, key sources of inferential
  leverage come from the research design and not
  from ex-post modeling (e.g., multivariate
  regression analysis, matching, etc.)
• Statistical techniques for analyzing experimental
  data are simple, which provides transparency

49
Experiments in the Study of Development

• To what extent can important topics in the study
  of development be approached with randomized
  controlled experiments?
• Some topics recently studied experimentally
• Clientelism/Vote Buying (effects of public
  information campaigns)
• Co-ethnic Cooperation (mechanisms that explain
  public goods provision)
• Deliberative democracy (the effects of
  development aid and the impact of leadership)
• At the same time, many key institutional
  variables are not manipulable by
  researcherswhich suggests the value of natural
  experiments

50
Natural experiments with as if randomization

• Key issues
• Are treatment and control groups unbalanced with
  respect to measured variables other than the
  treatment?
• Do units self-select into groups?
• Are policy interventions applied in a way that
  anticipates the behavioral response of units?
• No to all questions may be necessary, if not
  sufficient, for a credible natural experiment

51
Plausibility that assignment to treatment is
as if random
Most plausible
Least plausible
Snow (1855)
Brady et al. (2004)
Grofman et al. (1995)
Most observational studies
Randomized experiments
Card et al. (1994)
Posner (2004)
Angrist and Lavy (1999)
Doherty, Green, and Gerber (2004)
52
The role of qualitative methods

• Qualitative tools and methods are often
  invaluable and prerequisite for field and natural
  experiments
• Measuring outcomes, designing interventions,
  testing mechanisms
• The idea of experimental ethnography