Value Elicitation with

1
Value Elicitation with Experimental Auctions
Validity and Future Research Directions Jay
son L. Lusk Professor and Willard Sparks Endowed
Chair Oklahoma State University
2
Why Value Elicitation?

• Carry out cost/benefit analysis
• Determine welfare effects of technological
  innovation and public policy
• Better understand individual decision making and
  test/refine economic theory
• Improve firm marketing decisions related to
  pricing, advertising, and new product adoption

3
Why Experimental Auctions?

• Hypothetical surveys are known to generate biased
  results
• Auctions put people in an active market that, if
  properly designed, provides incentives for
  truthful value revelation
• Auctions provide information on individual
  WTP/WTA
• other value elicitation techniques (i.e.,
  discrete choice experiments) rely on statistical
  models and assumed functional forms to generate
  probability statements about valuations

4
Why Experimental Auctions?

• Auctions balance control and context
• evidence suggests values and behavior are context
  specific
• complete control can be self-defeating
• it creates an environment too sterile, too
  abstract, and too unreal, with no parallel in the
  real-world people cannot bring their learned
  experiences to bear on the task at hand
• too much context makes the experiment too much
  like an open-ended case study or descriptive
  naturalist observations
• experiments based on too many confounding factors
  yields no universal patterns and limits
  generalizability

5
Context Rich
Non-Experimental Data ex scanner data,
aggregate time series data
Value Elicitation Experiments I ex elicitation
of values for real goods in field setting
Value Elicitation Experiments II ex elicitation
of values for real goods in lab setting
High Control
Low Control
Value Elicitation Experiments III ex
elicitation of risk time preferences
Bargaining Experiments ex ultimatum game
Induced Value Experiments ex Smiths DOA
Context Poor
6
What Exactly is an Experimental Auction?

• In a homegrown value experimental auction,
  people either
• are endowed with a typical good and bid against
  N other people for a new good
• sometimes a consumption requirement is imposed
• bid against N other people to obtain a typical
  good or a new good with one good randomly
  selected as binding
• Market price and number of winning bidders is
  determined by an exchange mechanism that is
  theoretically incentive compatible

7
Incentive Compatible Auctions

8
What Exactly is an Experimental Auction?

• Bidding frequently takes place over several
  rounds with market price announced after each
  round and with one round randomly selected as
  binding
• Note the merits of many implementation issues
  (endowment vs. no endowment multiple rounds vs.
  single-shot BDM vs. 2nd price) are under debate
• in our forthcoming book (Cambridge University
  Press) Shogren and I discuss these issues and
  many more

9
Main Topics of Presentation

• Present evidence for/against the validity of
  experimental auctions as a method to measure
  valuations
• Identify 10 topics worthy of future research

10
Validity

• Validity is the extent to which a measurement
  instrument actually measures what it purports to
  measure
• We take economists notion of value as the
  theoretical construct of interest and we seek to
  determine the validity of experimental auctions
  in measuring these latent variables

11
Validity

• The goal is to explore whether auctions provide
  an accurate measure of value and whether elicited
  values respond in ways predicted by economic
  theory
• Why? Consider the following recent quote from a
  reviewer at a top economics journal
• My background is in psychology . . . I do not
  believe that economists should be using such
  auctions to study choice behavior and/or to
  elicit preferences that will be used to inform
  policy. Until such time as economists undertake
  rigorous, systematic comparisons of
  auction-revealed preferences and choices . . . in
  real markets, then people like me . . . will
  continue to ask for evidence of external
  validity.

12
Validity

• Consider the following recent quote from an
  editor at a top marketing journal
• . . . there is a difference between . . .
  whether auctions . . . can measure bidders true
  willingness-to-pay for an induced value
  laboratory experiment versus an auction for a
  real new good.
• Validity is addressed by investigating
• whether auction bids conform to predictions of
  economic theory
• reliability of auctions
• convergent validity
• how bids relate to taste tests, hedonic ratings,
  other value elicitation methods
• external validity

13
Economic Theory

• Does market price increase when demand increases?
• ?Yes, see Umberger and Feuz (2004)

WTP
Fixed supply of auctioned goods in 4th price
auction
0.2861.68
Demand curve from auction with 12 bidders
Demand curve from auction with 6 bidders
3
Q
14
Economic Theory

• Do bids reflect diminishing marginal utility?
• Do bids reflect non-satiation or scope?
• Yes, see Corrigan and Rousu (2006)

N 94
15
Economic Theory

• Is demand affected by the price/availability of
  substitutes and compliments?
• Yes, see Corrigan and Rousu (2006b)
• Mean bid to buy one unit of plain labeled corn
  chips was 0.51 and mean bid to buy one unit of
  US labeled corn chips was 0.58. Note
  0.510.581.09. The mean bid to buy both
  types of chips together was 1.03 i.e., the two
  are substitutes
• Mean bid to buy one unit of US labeled corn chips
  was 0.58 and mean bid to buy one unit of salsa
  was 0.65. Note 0.580.651.23. The mean
  bid to buy both chips and salsa was 1.31 i.e.,
  the two are compliments

16
Economic Theory

• Does WTP increase with information and decrease
  with information? Do people update their
  priors?
• Yes, see
• Within-subject studies Hayes et al. (1995),
  Hoffman, Lusk et al. (2004), Fox, Hayes, and
  Shogren (2000)
• Between-subject studies Hayes et al. (1995),
  Rousu et al. (2004), Huffman et al. (2004).

17
Economic Theory

• Is WTP/WTA affected by difficulty in delaying the
  decision and difficulty in reversing the
  transaction as predicted by the options value
  literature?
• Yes,
• Kling, List, and Zhao (2003) show WTP (WTA) is
  increasing (decreasing) in difficulty of delay
  and decreasing (increasing) in difficulty of
  reversing
• Kling, List, and Zhao (2003) also show WTP
  increases with a buy back guarantee
• Corrigan (2005) shows WTP is increasing in
  difficulty of delay and decreasing in difficulty
  of reversing

18
Economic Theory

• Is the value of 1.00 exactly 1.00?
• Yes,
• Corrigan (2005) had subjects bid to have 1.00 in
  a random nth price auction
• mean bid for the dollar bill was 0.97 and the
  median bid was 0.99

19
Reliability

• Reliability the extent to which repeated
  measures of value relate to one another
• reliability is a necessary but not sufficient
  condition for validity

20
Reliability

• Test-retest reliability (commonly used in
  psychology)
• Shogren, List, and Hayes (2000)
• Each subject participated in 4 experimental
  sessions over a two week time period
• The hypothesis of equality of mean bids across
  all four sessions could not be rejected in
  auctions for candy bars and mangos
• The only good for which bidding was not stable
  across sessions was irradiated pork, a more
  exotic good a finding they attribute to
  preference learning

21
Reliability

• Test-retest reliability
• Many auctions have person bid on the same good in
  several bidding rounds. Are bids correlated
  across round?
• Consider the data on bids for CAB steak from
  Lusk, Feldkamp, and Schroeder (2004) (n70)

22
Reliability

• Parallel-forms reliability are different,
  parallel measures are highly related? i.e., do
  different incentive compatible auctions give the
  same result?
• Evidence is mixed
• Rozan et al. (2004) bids from a BDM gt 2nd price
  auction
• Rutström (1998) English auction BDM lt 2nd
  price
• Lusk, Feldkamp, and Schroeder (2004) 2nd price
  random nth price English BDM in initial
  bidding rounds, but by final bidding rounds 2nd
  price gt English BDM gt random nth price
• Knetsch, Tang, and Thaler (2001) 2nd price gt 9th
  price

23
Reliability

• Even if one can reject equality of bids across
  mechanisms, are the differences economically
  important?
• Correlation between mean bids for 5 beef steak
  types across 4 mechanisms (n5)

24
Convergent Validity

• Convergent validity refers to whether measures
  that should be related are actually related

25
Convergent Validity

• Auction Bids and Taste Tests
• Several studies show bids for beef steaks,
  obtained after a taste test, are positively
  correlated with objective measures of beef steak
  tenderness Lusk et al. (2001a), Fuez et al.
  (2004), and Platter et al. (2005)
• Melton et al. (1996) and Platter et al. (2005)
  show bids for beef steaks and pork loins,
  obtained after a taste test, are positively
  correlated with marbling (fat content)

26
Convergent Validity

• Auction Bids and Hedonic Ratings
• In Lange et al. (2002), people evaluated 5
  different Champagnes in 3 different information
  conditions
• people either participated in an auction or a
  hedonic rating (0 certainly dislike, 10
  certainly like)
• the 5 drinks were similarly ranked in both
  methods and both methods responded similarly to
  new information
• Noussair, Robin, and Ruffieux (2004) used a
  within-subject design to compare hedonic ratings
  and auction bids for different varieties of
  orange juice, cookies, and chocolate bars
• On average, near identical preference rankings
  were obtained from auctions and ratings
• At the individual level, just over 60 of
  subjects have a weakly consistent ordering of
  varieties between the two measures about 75
  were consistent in identifying least and worst
  preferred varieties

27
Convergent Validity

• Auction Bids and Hedonic Ratings
• Wertenbroch and Skiera (2002) elicited bids to
  obtain Coca-Cola and Cake using a BDM mechanism
• bids were significantly correlated with responses
  to how thirsty/hungry are you right now and
  how much to you like Coca-Cola/cake
• correlations between bids and ratings were higher
  than the correlations between hypothetical
  statements of value and ratings
• Many other studies found similar results The
  only contrary result is in Jaeger and Harker
  (2005)
• found that although responses to two different
  hedonic scales taken after tasting two Kiwi
  fruits were significantly correlated with each
  other, neither were significantly related to bids
  for the fruit
• Note scales are often one-dimensional whereas
  bids reflect values for numerous attributes

28
Convergent Validity

• Auction Bids and Other Measures of Value
• Many studies show bids from auctions lt
  hypothetical valuations (auctions, CVM, etc.),
  but measures are often correlated
• Mixed evidence comparing valuations from
  non-hypothetical choice to auction bids
• Lusk and Schroeder (2006) WTP from choice gt bids
• Frykblom and Shogren (2000) WTP from choice
  bids
• Ding, Grewal, and Liechty (2005) choice behavior
  ? bids

29
External Validity

• How well do auction bids relate to actual retail
  behavior?
• Ding, Grewal, and Liechty (2005) used BDM bids to
  forecast a choice of which Chinese dinner to eat
• the auction mechanism performed 3x better than a
  naïve prediction and 2x better than a
  hypothetical open-ended pricing task
• Brookshire, Coursey, and Schulze (1987) compared
  demand curves constructed from bids for
  strawberries collected in a laboratory auction to
  implied demand curves from actual purchases of
  strawberries made via door-to-door sales
• They were unable to reject the hypothesis that
  the valuations from the auction were different
  than the field sales data

30
External Validity
Highest and 2nd highest bids for 12 sports cards
compared to published book values source List
and Shogren (1998)
31
Validity

• Overall, results support the notion that
  experimental auctions are valid
• Some evidence exists to contradict this view
• Such findings often only imply a weakness in a
  particular implementation of the method, not
  necessarily with the entire valuation paradigm

32
Validity

• There are other anomalies presented as a
  challenge to the validity of experimental
  auctions and economists notion of values
• In some cases, repeated experiments and
  development of economic theory have served to
  address some such challenges
• In other cases, we do not have enough information
  to understand under what conditions economic
  theory and auction valuations fail and how
  economic theory might evolve to address the
  concerns
• Even in the instances in which violations of
  individual rationality have been observed, other
  studies provide evidence to suggest that the
  power of auctions and the discipline of markets
  have the potential to generate rational behavior

33
10 Questions for Future Research

• Do experimental auctions elicit rational
  statements of value?
• relying on rational theory to guide valuation and
  policy makes more sense if people make, or act as
  if they make, consistent and systematic choices
  future work should address the circumstances
  under which the presumption of rationality is and
  is not supported
• Do experimental auctions mimic real world
  behavior? Do they exhibit external validity?

34
10 Questions for Future Research

• How well do experimental auctions forecast retail
  behavior relative to other research methods?
• even if one could reject external validity of
  experimental auctions could another method do any
  better?
• How do experimental auctions compare to
  incentive-compatible, choice-based value
  elicitation methods and can the methods be
  combined to improve estimates of peoples values?
• both methods have strengths and weaknesses

35
10 Questions for Future Research

• Can experimental auctions be more broadly
  employed to test economic theory?
• experimental auctions provide detailed
  information about preferences without requiring
  strong a priori assumptions about utility
  functionals
• e.g., tests of consumer theory WARP,
  homogeneity, symmetry, separability, etc.
• Can debates over design issues in experimental
  auction methods be settled with non-experimental
  data?
• e.g., do auctions with endowments forecast retail
  behavior better than auctions that do not use
  endowments?

36
10 Questions for Future Research

• How useful are experimental auctions for pattern
  recognition?
• development of empirically-based theories
• e.g., hypothetical bias Plott-Zeiler study of
  WTP/WTA
• How do personality traits affect bidding behavior
  in experimental auctions?
• standard big 5 personality traits
• other interesting traits competitiveness,
  concern for social standing, culture

37
10 Questions for Future Research

• How do emotions and auction behavior interact?
• How do emotions affect valuations? ex Lerner et
  al. (2004) showed that if people experienced the
  emotion of disgust that WTP measures of value can
  exceed WTA measures of value.
• How do economists model multiple emotions?
• Can experimental auctions predict the future?
• ex bid to buy a contract that pays 10 if event
  X happens before end of year
• simplified versions of prediction markets used
  to forecast election outcomes but perhaps more
  informative as each individuals expectation is
  elicited

38
Conclusions

• Much has been learned about the pros and cons of
  experimental auctions to elicit demand for new
  goods and services
• Researchers understand better now how people
  learn about and react to incentives,
  institutions, and information
• More work is needed to refine designs and develop
  the method so that we can better understand
  consumer behavior in an effort to improve
  business decision making and public policy

39
Conclusions

• If you want to learn about the method, run
  an experimental auction yourself. Inventing and
  implementing an experiment is still the best way
  to discover the power and limits of the method.
  Ask the question you are interested in, study
  previous attempts to address similar questions,
  design your experimental auction, look for a
  balance of control and context, think about
  options outside the lab since we cannot create a
  perfect people vacuum, think about incentives
  in light of intrinsic and extrinsic motives,
  prepare to explain yourself to others who might
  have wanted you to ask a different question, have
  fun, and get ready to do it again because you
  might end up finding more questions than answers.