EITM: Experimental Implications of Theoretical Models

1
EITM Experimental Implications of Theoretical
Models

• Guest Lecturer Rebecca Morton New York
  University

2
Plan for Today . . .

• What Experimentation is (and is not) generally
• What Formal Theory Testing is (and is not)
  generally
• Experimentation Formal Theory Testing in
  Political Science
• Trying out Experimentation . . .

3
What is Experimentation?

• Fundamentals of Experimentation
• Manipulation
• Control (baseline, comparisons)
• Random Assignment
• Variation in these across disciplines within

4
Experiments v. Simulations

• real choices and real decisions independent
  of researcher.
• computer simulations extension of researchers
  brain not real data like experiments.

5
Experimental v. Non-exp. Data

• In design process choose which variables expect
  important and control values of independent
  variables before data generated.
• Not possible using survey data archival data
• Can generate data that may be impossible or
  extremely difficult to observe non-experimentally.

6
Natural Experiments

• Example Kenny Lott womens suffrage
• Not an experiment
• Why? What is missing?
• Random Assignment!

7
Random Assignment

• Advances in statistics of late 19th early 20th
  centuries important for experiments
• Can eliminate, within statistical limits,
  extraneous factors that obscure effects expect to
  observe.

8
Random Assignment

• Non-experimentalist makes statistical assumptions
  about factors often untestable.
• Results only as good as statistical assumptions.
• Experimentalist uses random assignment.

9
Types of Experiments

• Laboratory or Field?
• Audience or Goal Theory, Facts, or Policy?
• Roth
• speak to theorists,
• search for facts,
• or whisper in the ears of princes
• Level of Analysis Individual or Group?
• Motivational Incentives Financial or Not?
• Script or Not?
• Computer or Not?

10
All of These Types of Experiments are Increasing
in Political Science
11
Why Now?

• Psychology in late 1800s.

• 1924 Harold Gosnell conducted field experiment
• Political science adopted other behavioral
  methods in 50s 60s, but not experiments.

12
Yet, Political Science Lost Experiments Like
America was lost to Europeans after Vikings

• Not until 1970s see labs Stony Brook Caltech
• Christopher Columbuses?

13
Why Experiments Delayed

• A discipline becomes experimental when variables
  of theoretical interest are susceptible to
  experimental control.
• Examples
• physics after Newton others created theoretical
  concepts (such as force, mass, etc.) suitable for
  controlled manipulation.
• psychology when switched from studying
  consciousness, using introspection to
  studying behavior, using experiments.

14
Why Experiments Delayed

• Political science study of specific real
  world behavioral domain (unlike psychology)
• Political scientists traditionally reasonably
  worry about artificiality.
• Experimentation, by introducing artificiality is
  suspect.

15
Why Experiments Have Reappeared

• Inability of Existing Data to Answer Important
  Causal Questions
• New Research Questions

16
Inability of Existing Data

• Example influence of tv on American public
  opinion?
• Surveys cannot establish causal relationship
• Iyengar Kinder 1987 use experiments

17
New Research Questions

• Underpinnings of political behavior
  implications for observed behavior
• How voters process information during election?
  (work of Lodge, et al, Lau etc.)
• (theory assumption testing -- more on this
  shortly)

18
New Research Questions

• Evaluation of Proposed Reforms
• e.g. cumulative voting, sequential voting

19
Is Rise in Abstraction Culprit?

• Econ embraced abstraction not experimentation
• interested in testing consequences rather than
  underlying process
• now changed. Why?
• Rise of game theory w/ multiple equilibrium
  predictions new questions about process

20
Theory Experimental Design, Part I
21
First Question Goal?

• Types of Goals
• Theory Testing
• Fact Finding
• Policy Pre-testing

22
Theory Testing What Type?

• FTT Formal Theory Testing
• PTT Psychological Theory Testing
• ATT A(nti)Theory Testing

23
Digression My View of FTT (from Methods
Models)
24
Terms

• Nonformal Model
• Formal Model
• Empirical or Statistical Model

25
Model Building . .
26
Terms

• Mathematical Model
• Computational Model
• Pure Theory
• Applied Formal Model

27
(No Transcript)
28
(No Transcript)
29
Methods of Empirical Evaluation
30
Types of Empirical Evaluation

• Assumption Evaluation
• Prediction Evaluation
• Equilibrium Predictions
• Equilibrium Point Predictions
• Disequilibrium
• Multiple Equilibrium
• Relationship Predictions
• Comparative Static Predictions
• Dynamic Path or Process Predictions
• Alternative Model Comparisons

31
Assumption Evaluation

• Pure Theory Sometimes silly test Assumptions
  not point of exercise
• Applied Formal Theory
• Should evaluate, consider
• But NOT substitute for theory evaluation (part of
  process)

32
Assumption Evaluation

• What does Assumption Testing Mean?
• If find false two possibilities
• If relaxed qualitative results of model still
  follow.
• Results hinge crucially on false assumption
  further empirical theoretical work necessary
• If find true must remember
• True only for conditions under which tested
• Still must assume true for theory situation

33
Assumption Evaluation

• Lead further theorizing.
• Dangers of Quest for realness
• Too complex models (what causes what????)
• Hurt Pure Theory Efforts

34
(No Transcript)
35
Evaluating Relationship Predictions

• What learn?
• support or not for model
• size of relationship
• Empirical Model v. Theoretical Model
• Empirical model often more restrictive than
  underlying model (specification estimation)
• Adding control variables random error means
  empirical evaluation is different (partial DGP)

36
Evaluating Alternative Models

• Really independent alternatives?
• Rare get conclusive analysis
• Multiple equilibria makes overall fit comparison
  hard
• Assumptions or predictions tests, neither
  sufficient
• Look for KEY differences, testable.
• Empirical models assumptions may bias in favor
  of one model

37
Laboratory Experiments Fantastic Method of
Testing Formal Theories
38
Difficulties w/ Naturally Occurring Data

• Seldom find empirical situations where ONLY
  variables of theoretical interest are active
• Go to Partial DGP or complex model
• Problem of measuring variables of theoretical
  interest (utilities, beliefs) in natural
  settings.
• Assume functions of what can measure (I.e. voting
  choices, survey answers, v. true voter
  preferences)

39
Experimental Formal Theory Testing (FTT)

• How work? Idea is to treat theory as script
  for experiment.
• High internal validity

40
Formal Theory Testing (FTT)

• Ideal experimental analysis like structural
  estimation of a formal model with naturally
  occurring data
• In structural estimation estimating equations
  used on naturally occurring data (usually a
  maximum likelihood function) directly derived
  from formal model

41
Why Internal Validity Matters

• Modal Empirical Work Partial DGP (Regress)
  using naturally occurring data
• Theorists discount results
• Not really testing model quibble about
  variables chosen, controls, econometric
  techniques, disconnects between theory empirics
• Experimental FTT (Progress)

42
How FTT Works

• Reproduce conditions of theory in laboratory
• One subject for each agent in theory
• Give subjects same incentive as agents in theory
• See how agents behave.

43
Lots of FTT Already Accomplished

• Huge literatures (in econ, sociology, psychology,
  a little political science) on classic games
  such as
• public good games
• Prisoners Dilemma games
• bargaining (ultimatum, dictator)
• cooperative games (battle of the sexes)
• Simple market experiments (buying selling a
  good)

44
Lots of FTT Already Accomplished

• Growing literature on some not so obviously
  classic games (centipede, beauty contest)
• An extensive literature on decision-theoretic
  experiments designed to test rationality
  assumptions like expected utility etc.
• Much reviewed in Kagel, J. H. and Roth, A. E.
  (1995), Handbook of Experimental Economics,
  Princeton U.P.

45
Classic FTT Experiments

• Why study classic FTT Exp. from other
  disciplines?
• Often building blocks for formal models in poli
  sci
• Good examples of FTT
• Thus examine few before turning to political
  science examples.

46
An Example of a Prisoners Dilemma Game
47
General Structure of PD Gamea gt c d gt b
b gt a
48
PD Games Theory

• In one shot game, expect defection
• With repetition infinitely repeated many
  possible equilibria (folk theorem) including
  cooperation
• Axelrods work is it an experiment?

49
PD Games Experiments

• See Kagel, J. H. and Roth, A. E. (1995), Handbook
  of Experimental Economics, Princeton U.P., pages
  26 to 28 variations on the PD have been the
  subject of virtually continuous experimental
  interest since the 1950 experiment of
• Always some co-operation even in finite games
• Extent of co-operation first grows...
• then diminishes
• but does not disappear entirely

50
Public good problems

• Simplest form
• one private good - price 1
• one public good - price p
• all can consume public good
• question is who pays for it?

51
Stark form

• n players i 1, 2, ,n
• all get income z
• each decide how much x to consume privately, and
  how much y to contribute to the public good,
  where x y z
• payoff/utility to i xi (y1.. yi..yn)/p
  xi (z1-x1)(zi-xi)(zn-xn)/p

52
Optimal choices

• privately to max
  xi (z1-x1)(zi-xi)(zn-xn)/p
  put xi m and yi 0 (assuming p gt 1)
• private payoff/utility m for all
• publicly put xi 0 and yi m
• private payoff/utility nm/p
• if n/p gt 1 then private optimal worse
• i.e. 1ltp lt n private optimising bad for all

53
Early Experimental Work

• Marwell Ames found most subjects contributed
  at a point between 40 60 between private
  public optima.
• Variables investigated
• size of group (n)
• public good price (p)
• repetition (deception used by some experimenters)

54
Effect of Repetition

• Usual finding - contributions to public good
  start out between 40 60 of difference between
  private public optima
• Contributions declined with repetition
• But rarely declined to private optimum
• WHY?

55
Survey of Early Work

• Ledyard, J.O. (1995), Public Goods A Survey of
  Experimental Research, in Kagel, J. H. and Roth,
  A. E. (1995), Handbook of Experimental Economics,
  Princeton U.P.
• HUGE literature on classic games like this,
  almost everything you can think of has been tried
  . . .

56
PD Public Good Types of Tests

• Equilibrium predictions
• fails for public goods, succeeds in some simple
  market experiments, dictator games
• Relationship predictions
• in public good game varying n p likelihood of
  contributing succeeds
• Non-predictions (fact finding)
• repetition, communication, types of subjects,
  varying parameters, framing, many such things

57
FTT Exp v. Traditional ExperimentalMethods

• Recall fundamentals of experimentation
• Manipulation
• Control (baseline)
• Random assignment
• Originally test simple theories

58
Traditional Experimental Method

• Test hypothesis drinking diet coke before driving
  leads to less safe driving.
• Have subjects drive test course, randomly having
  some drink a diet coke before driving, others not
  given anything to drink in advance
• Random assignment means that differences between
  subjects in safeness can be considered
  independent.
• Manipulate through providing the diet coke
• Baseline or control those not getting diet coke

59
FTT Tradition

• Entire experiment takes theory as script
  baseline experiments are used as comparisons in
  testing relationship predictions of theory.
• Subjects randomly assigned to treatments,
  however, treatments are all within theorys
  script as given by formal model.
• Manipulation, control, random assignment
  manifest differently because of nature of theory.

60
Formal Theory Testing (FTT) and Fact Finding

• Note relationship between FTT fact finding
• I.e. theory fails (or maybe succeeds), but maybe
  some things not in theory matter, lets vary
  them.
• Also called Stress Tests
• Fundamentally different from Partial DGP w/
  naturally occurring data (Progress not Regress)
  CONTROL

61
Examples of Theory-Driven Fact Finding

• Examples
• Cooperation in Public-Goods Experiments
  Kindness or Confusion, James Andreoni, American
  Economic Review, Vol. 85, No. 4. (Sep., 1995),
  pp. 891-904.
• Bargaining and Market Behavior in Jerusalem,
  Ljubljana, Pittsburgh, and Tokyo An Experimental
  Study, Alvin E. Roth, Vesna Prasnikar, Masahiro
  Okuno-Fujiwara, Shmuel Zamir, American Economic
  Review, Vol. 81, No. 5. (Dec., 1991), pp.
  1068-1095.

62
Theory Fact Finding New Theory More Fact
Finding, etc.

• Examples see recent theoretical research on
  learning fairness which has been used to
  explain experimental results in classic games
• Combining across experiments Gary Bolton,
  Bargaining and Dilemma Games From Laboratory
  Data Towards Theoretical Synthesis, Experimental
  Economics, 1257-281(1998)

63
Design Issues FTT Experimenters Faced Discuss
in Detail Shortly

• Repetition practice? Is this important or not?
  FTT usually (but not always) have subjects
  perform tasks repeatedly
• Scripts (instructions)? How specific should they
  be? How should experiment be framed? FTT
  usually uses very specific scripts, designed to
  capture details of theory institutions
• How should subjects be motivated? FTT usually
  uses financial incentives dependent on subjects
  choices to motivate.

64
Some Design Issues Early Experimenters Faced
Computers

• Early FTT experiments conducted by hand, before
  PC computer labs.
• Time consuming . . .
• Subject boredom
• Advent of Computers
• Complex Designs Possible
• Belief problems
• Software Problems

65
Experiments on Not So Classic Games
Rationality Assumptions

• special games arent obviously capturing reality
  but strong tests of theoretical concepts.
• Centipede game
• Beauty Contest
• Large extensive literature (going back to 1950s)
  testing rationality assumptions proposed
  alternatives (decision-theoretic experiments)

66
Four move centipede game
67
Centipede Games

• Subgame perfect equilibrium is to end at first
  move.
• Yet would not expect subjects to do this, they
  dont.
• How far will they go?
• Interesting game to consider explanations for why
  subgame perfection might fail.
• Research has had a big perhaps surprisingly
  influence on political science as will see
  later.

68
Beauty Contest Games

• professional investment may be likened to
  those newspaper competitions in which the
  competitors have to pick out the six prettiest
  faces from a hundred photographs, the prize being
  awarded to the competitor whose choice most
  nearly corresponds to the average preferences of
  the competitors as a whole. (Keynes, GT,
  155-156)

69

• It is not a case of choosing those which, to
  the best of ones judgement, are really the
  prettiest, nor even those which average opinion
  genuinely thinks the prettiest. We have reached
  the third degree where we devote our
  intelligences to anticipate what average opinion
  expects the average opinion to be. And there are
  some, I believe, who practise the fourth, fifth
  and higher degrees. (Keynes, GT, 155-156)

70
Literature

• Nagel, R., (1995), Unravelling in Guessing
  Games An Experimental Study, American Economic
  Review, 85, 1313-1326.
• Ho, T. H., Camerer, C. and Weigelt, K., (1998),
  Iterated dominance and iterated best response in
  experimental "p-beauty contests", American
  Economic Review, 88, 947-969

71
Basic Beauty Contest Game

• Large number M of people choose simultaneously a
  number in 0,100
• Winner is person whose number is closest to mean
  multiplied by p
• p positive pre-determined
• Winner gets pre-determined prize which is shared
  equally if there is a tie.

72
How Would You Play This Game?

• Let p 1/2 so winner is person whose chosen
  number is nearest to one half the average of all
  chosen numbers
• Everyone write down a number between 0 100 on
  piece of paper
• Prize is a (hypothetical) 100.

73
Game theoretic solutions

• if 0 lt p lt 1 there is unique Nash equilibrium
  when all players choose 0
• if p 1 (if M gt 2) infinitely many equilibria
  (becomes a co-ordination game)
• if p gt 1 and 2p lt M two equilibria all choosing
  0 and all choosing 100

74
Boundedly rational 1st period play

• Strategic of degree 0 if chooses 50
• strategic of degree 1 if chooses 50p
• strategic of degree 2 if chooses 50p2
• strategic of degree n if chooses 50pn

75
Boundedly rational later period play

• Let r be the mean on the previous play
• Strategic of degree 0 if chooses r
• strategic of degree 1 if chooses rp
• strategic of degree 2 if chooses rp2
• strategic of degree n if chooses rpn

76
Beauty Contests Results

• 1st-period choices widely distributed far from
  equilibrium.
• But subsequent choices converge towards
  equilibrium.
• 1st-period choices consistent with a median of 2
  steps or 1 step iterated dominance (depending on
  design).
• Choices after 1st consistent with 70 of Subjects
  best responding.
• Play is sensitive to p, M experience.

77
Beauty Contests Conclusions

• Subjects unable to move directly to equilibrium.
• Depth of reasoning displayed depends upon
  experience, type of game individual.
• Some convergence but also some stickiness away
  from equilibrium observed.

78
Assumption Testing Decision-Theoretic
Experiments on Rationality

• FTT Exp. Work that examines assumptions about
  individual decision making underlying formal
  models in simple decision-making experiments
  rather than in context of a game.

79
Literature Extensive

• Early challenges to expected utility theory in
  1950s 1960s to current period, see review by
  Colin Camerer in Handbook.
• Often no financial motivations contingent on
  choices, although growing of experiments w/
  choices w/ financial motivations.
• Typically simple choices to see if basic
  preference assumptions are violated, such as
  independence axiom (see discussion MM).

80
Political Science Assumption Testing

• Most aware of preference reversal experiments of
  Kahneman Tversky prospect theory substitute.
• a few similar experiments in political science.
• Most decision-theoretical experiments in
  political science not explicitly connected to
  formal theories or their assumptions typically
  based on non-formal theories.