How close should experiments be to models and why

1
How close should experiments be to models and why?

• Assessing Experimental Economics(??)
• by N.Bardsley, R.Cubitt, G.Loomes, P.Moffatt,
  C.Starmer and R.Sugden, forthcoming
• Economics experiments tend to resemble formal
  models
• Officially, this allays external validity worries
• Internal validity extent to which the lab
  results inform us about whats happening in the
  lab
• External validity extent to which the lab
  results inform us about aspects of the world at
  issue
• Therefore Internal validity is logically prior

2
The Blame the Theory Defence

• Vernon Smith (1982) what is most important to
  any particular experiment is that it be relevant
  to its purpose. If its purpose is to test a
  theory, then it is legitimate to ask whether the
  elements of alleged unrealism in the experiment
  are parameters in the theory. If they are not
  parameters of the theory, then the criticism of
  unrealism applies equally to the theory and the
  experiment.
• In other words if we set up experiments to
  resemble economic models, we give theories their
  best chance. So we can ignore external validity
  if our aim is to test a theory.
• But consider an example from scientist James
  Lovelock

3
Prof. David M. Bice, Department of Geology,
Pennsylvania State University
http//www.carleton.edu/departments/GEOL/DaveSTELL
A/Daisyworld/daisyworld_model.htm
4
Economic Models as Claim-Ready Sets of
Propositions

• Theory Tx (compound) claim T about aspect of
  the world x
• Tx mCx where m is (usually) a set of
  propositions with semantic content appropriate to
  x
• m is expounded as a set of assumptions or
  stipulations, not assertions about the world
• C is a predicate relating the model to the world
• C might be realist, as-if, instrumentalist or
  parable-relating
• C, and therefore Tx, is often oblique or implicit
• Cf. Lovelocks explicit Daisyworld rationale to
  demonstrate that purposeful action by nature is
  not necessary for life to regulate the Earths
  temperature. This could work by natural
  selection.
• Daisyworld is a prop for Gaia theory, which does
  not refer to Daisies as a mechanism at all.

5
Consequences

• To test a theory by physically setting up the
  model is absurd Gaia theory is a parable about
  the Earth!
• Whiteheads (1925) fallacy of misplaced
  concreteness
• Whether an experimental environment E tests a
  theory depends whether Tx predicts for it
• This depends on the presence of x, not m that E
  resembles m says nothing about this.
• It doesnt require intended instances of x
  reference may be unintended

6
The Validity Problem

• The artificiality of alteration versus that of
  contamination or isolation (Greenwood 1982)
• Relationality of social phenomena means Tx
  often does not make predictions for the lab
• E.g. a jury trial requires a judge, and that
  participants recognise the judges formal
  authorisation etc.
• The experimenters experiment may diverge from
  the subjects experiment (Orne 62, 73) internal
  (and so external) validity requires that the two
  converge
• Its the subjects experiment(s) that drives
  behaviour
• Examples Tax evasion experiments, dictator games

7
Tax Compliance and Evasion

• pi ei tdi Im(1-di) I 1 or 0
• e.g. Alm et al. 1992 claims evidence that people
  dont pay taxes out of moral / citizenship
  obligation
• But a tax is a collection of revenue by the
  government, a specific authority authority is
  relational
• The data are on monetary gambles nothing else is
  there
• Ockhams razor alternative for taxation
  (insurance c) inferences
• Reduction E.g. suppose an experiment finds
  dR/dt gt 0 R tSidi
• But the real tax-revenue relationship may vary
  with the perceived legitimacy of the government
  or tax
• Seen as Legitimate dR/dt gt 0 Illegitimate
  dR/dt lt 0
• So problems of causal holism are intertwined with
  relationality
• Tax-revenue-government-electorate-citizenship

8
Dictator Game
Bardsley (2008) Dictator Game Giving Altruism or
Artefact? Experimental Economics, 11 (2)
9
Taking Game
Bardsley (2008) Dictator Game Giving Altruism or
Artefact? Experimental Economics, 11 (2)
10
Challenge Demand Characteristics

• Experimentalists Cant dismiss results of an
  experiment because DCs might be a problem
• Sceptic DC confounds might be frequent
• Cf Placebo effects in medicine
• Observations
• DCs seem plausible in some cases, not in others
• Need for theorising about where DCs are likely to
  be (un-)problematic (e.g. inscrutable
  hypothesis?)
• Need for empirical checking for DCs (à la Orne)

11
Theories and Domains (Cubitt 2005)
Testing Domain
Base Domain
Intended Domain
12
Theories and Domains (Cubitt 2005)
Testing Domain
Base Domain
Intended Domain
13
Theories and Domains
x in mCx
Testing Domain
Base Domain
Intended Domain
14
Reflections on BTT and Falsificationism

• BTT may still be valid if the experiment does
  implement x in Tx
• E.g. most experiments on individual choice theory
  locate the design in the base domain of the
  theory, by using well-defined probabilities and
  outcomes for choices. Arguably, hyothetical
  designs (psychology) do not.
• But without induction, we must say that failure
  of the theory in the lab is no guide to its
  future performance either in the field or the
  lab!
• Empirical work refines theories. Falsificationism
  is blind to this
• If we always give the theory its best chance
• 1. We may end up inductively refining our
  theories around phenomena that are externally
  rare, getting locked into our own little
  world(s).
• 2. Success of the theory in the lab might be
  uninformative about its performance in the field.
• Therefore even where BTT is valid for an
  individual design there are good reasons to be
  doing less model-constrained work.

15
Validity of BTT defence

• Naïve Experimental Claim
• Any laboratory environment E in the base domain
  of a theory should be presumed to be in the
  T-domain
• Modified Experimental Claim
• unless there is some difference between E and the
  I-domain, which can reasonably be expected to
  make behaviour in the I-domain markedly more
  consistent with the theory (Cubitt, 2005,
  Bardsley et al. 2008).
• Proposed further modification
• or in the case of bold, Popperian testing, which
  can reasonably be expected to make behaviour in
  the I-domain less consistent with the theory
  (proposed further modification)
• NB E is in base domain only if Tx predicts for E
• Implementation of m is neither necessary nor
  sufficient

16
Is there a trade-off between EV and IV?

• local vs. broader EV IV?local EV, absent demand
  problems
• There is a trade-off between internal and
  external validity involving conditional
  probability
• Own little world designs may score highly on IV
  and weakly on broader EV
• Designs closer to naturally-occurring situations
  make internal validity harder to achieve e.g.
  free communication erodes statistical
  independence. But conditional on achieving it we
  may be more confident that the results hold
  outside the lab.
• This trade-off is about harnessing local EV to
  induct to situations of interest. Justifying
  claims to broader external validity is
  problematic even in the philosophy of natural
  science (Bruno Latour, Nancy Cartwright,
  Francesco Guala).
• Very local EV has no meaning in
  model-implementing Applied Economics designs,
  literally interpreted.

17
Conclusions How close should experiments be to
models and why?

• Theres no requirement for designs to resemble
  models for testing, the requirement is for the
  theory to predict for the design
• Proximity to a model may reduce internal ( so
  external) validity, since implementing the model
  may miss the target of Tx entirely
• Where designs do implement phenomena of interest,
  one may wish to give the theory its best chance
  by sticking to its base domain
• But thats just one experimental strategy, we
  need more to break out of own little worlds
  let a thousand flowers bloom!
• Evidential claims of model-implementing AE
  designs are suspect unless couched as
  decision-theoretic and analogical
• Such analogical claims may be unconvincing
  because of their reductive nature, and
  differences between human subjects and target
  entities

18
References

• Alm, J. McClelland, G.H. and Schulze, W.D. 1992.
  Why do people pay taxes? Journal of Public
  Economics, 48, 21-38.
• Bardsley, Nicholas 2008. Dictator game giving
  altruism or artefact? Experimental Economics, 11,
  122-133.
• Bardsley, N., Cubitt, R., Loomes, G., Moffatt,
  P., Starmer, C. and Sugden, R. 2008. Assessing
  Experimental Economics (working title),
  forthcoming.
• Cubitt, Robin P. 2005. Experiments and the domain
  of economic theory, Journal of Economic
  Methodology, 12, 197-210.
• Greenwood, J.D. 1982. On the relation between
  laboratory experiments and social behaviour
  causal explanation and generalisation, Journal of
  the Theory of Social Behaviour, 12, 225-249.
• Lovelock, J. 2005. Gaia Medicine for an Ailing
  Planet. Gaia books.
• Orne, M.T. 1962 On the social psychology of the
  psychological experiment with particular
  reference to demand characteristics and their
  implications, American Psychologist, 17, 11,
  776-783.
• Orne, M.T. 1973 Communication by the total
  experimental situation in P. Pliner, L.Krames and
  T. Alloway (eds.) Communication and Affect, 2nd
  edition, 157-191, New York Academic Press.
• Smith, V.L. 1982. Microeconomic systems as an
  experimental science, American Economic Review,
  72, 923-955.
• Whitehead, A.N. 1925 (1919). An Enquiry
  concerning the Principles of Natural Knowledge
  2nd ed. Cambridge University Press.