Title: Prediction%20Markets%20and%20the%20Wisdom%20of%20Crowds
1Prediction Markets and the Wisdom of Crowds
- David Pennock, Yahoo! Research
- Joint with
- Yiling Chen, Varsha Dani, Lance Fortnow, Ryan
Fugger, Brian Galebach, Arpita Ghosh, Sharad
Goel, Mingyu Guo, Joe Kilian, Nicolas Lambert,
Omid Madani, Mohammad Mahdian, Eddie Nikolova,
Daniel Reeves, Sumit Sanghai, Mike Wellman, Jenn
Wortman
2Bet Credible Opinion
Obama will win the 2008 US Presidential election
I bet 100 Obama will win at 1 to 2 odds
- Which is more believable?More Informative?
- Betting intermediaries
- Las Vegas, Wall Street, Betfair, Intrade,...
- Prices stable consensus of a large number of
quantitative, credible opinions - Excellent empirical track record
3A Prediction Market
- Take a random variable, e.g.
- Turn it into a financial instrument payoff
realized value of variable
Bin Laden captured in 2008?(Y/N)
I am entitled to
Bin Ladencaught 08
Bin Ladencaught 08
1 if
0 if
4http//intrade.com
5Outline
- The Wisdom of Crowds
- The Wisdom of Markets
- Prediction MarketsExamples Research
- Does Money Matter?
- Combinatorial Betting
Story
Survey
Research
Research
6A WOC Story
Story
1/7
Survey
Research
Opinion
- ProbabilitySports.com
- Thousands of probability judgments for sporting
events - Alice Jets 67 chance to beat Patriots
- Bob Jets 48 chance to beat Patriots
- Carol, Don, Ellen, Frank, ...
- Reward Quadratic scoring ruleBest probability
judgments maximize expected score
7Individuals
- Most individuals are poor predictors
- 2005 NFL Season
- Best 3747 points
- Average -944 Median -275
- 1,298 out of 2,231 scored below zero(takes
work!)
8Individuals
- Poorly calibrated (too extreme)
- Teams given lt 20 chance actually won 30 of the
time - Teams given gt 80 chance actually won 60 of the
time
9The Crowd
- Create a crowd predictor by simply averaging
everyones probabilities - Crowd 1/n(Alice Bob Carol ... )
- 2005 Crowd scored 3371 points(7th out of 2231)
! - Wisdom of fools Create a predictor by averaging
everyone who scored below zero - 2717 points (62nd place) !
- (the best fool finished in 934th place)
10The Crowd How Big?
Morehttp//blog.oddhead.com/2007/01/04/the-wisdo
m-of-the-probabilitysports-crowd/http//www.overc
omingbias.com/2007/02/how_and_when_to.html
11Can We Do Better? ML/Stats
Dani et al. UAI 2006
- Maybe Not
- CS experts algorithms
- Other expert weights
- Calibrated experts
- Other averaging fns (geo mean, RMS, power means,
mean of odds, ...) - Machine learning (NB, SVM, LR, DT, ...)
- Maybe So
- Bayesian modeling EM
- Nearest neighbor (multi-year)
12Can we do better? Markets
13Prediction MarketsExamples Research
14The Wisdom of CrowdsBacked in dollars
- What you can say/learn chance that
- Obama wins
- GOP wins Texas
- YHOO stock gt 30
- Duke wins tourney
- Oil prices fall
- Heat index rises
- Hurricane hits Florida
- Rains at place/time
- Where
- IEM, Intrade.com
- Intrade.com
- Stock options market
- Las Vegas, Betfair
- Futures market
- Weather derivatives
- Insurance company
- Weatherbill.com
15Prediction MarketsWith Money Without
16The Widsom of CrowdsBacked in Points
- HSX.com
- Newsfutures.com
- InklingMarkets.com
- Foresight Exchange
- CasualObserver.net
- FTPredict.com
- Yahoo!/OReilly Tech Buzz
- ProTrade.com
- StorageMarkets.com
- TheSimExchange.com
- TheWSX.com
- Alexadex, Celebdaq, Cenimar, BetBubble,
Betocracy, CrowdIQ, MediaMammon,Owise,
PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds - http//www.chrisfmasse.com/3/3/markets/Play-Money
_Prediction_Markets
17http//betfair.com
Screen capture 2008/05/07
http//tradesports.com
Screen capture 2007/05/18
18Example IEM 1992
Source Berg, DARPA Workshop, 2002
19Example IEM
Source Berg, DARPA Workshop, 2002
20Example IEM
Source Berg, DARPA Workshop, 2002
21Does it work?
Thanks Yiling Chen
- Yes, evidence from real markets, laboratory
experiments, and theory - Racetrack odds beat track experts Figlewski
1979 - Orange Juice futures improve weather forecast
Roll 1984 - I.E.M. beat political polls 451/596 Forsythe
1992, 1999Oliven 1995Rietz 1998Berg
2001Pennock 2002 - HP market beat sales forecast 6/8 Plott 2000
- Sports betting markets provide accurate forecasts
of game outcomes Gandar 1998Thaler
1988Debnath EC03Schmidt 2002 - Laboratory experiments confirm information
aggregationPlott 198219881997Forsythe
1990Chen, EC01 - Theory rational expectations Grossman
1981Lucas 1972 - Market games work Servan-Schreiber 2004Pennock
2001
22Prediction MarketsDoes Money Matter?
23The Wisdom of CrowdsWith Money Without
24The Wisdom of CrowdsWith Money Without
25Real markets vs. market games
HSX
FX, F1P6
forecast source avg log score F1P6 linear
scoring -1.84 F1P6 F1-style scoring -1.82 betting
odds -1.86 F1P6 flat scoring -2.03 F1P6 winner
scoring -2.32
26Does money matter? Play vs real, head to head
- Experiment
- 2003 NFL Season
- ProbabilitySports.com Online football forecasting
competition - Contestants assess probabilities for each game
- Quadratic scoring rule
- 2,000 experts, plus
- NewsFutures (play )
- Tradesports (real )
- Used last trade prices
- Results
- Play money and real money performed similarly
- 6th and 8th respectively
- Markets beat most of the 2,000 contestants
- Average of experts came 39th (caveat)
Electronic Markets, Emile Servan-Schreiber,
Justin Wolfers, David Pennock and Brian Galebach
27(No Transcript)
28Does money matter? Play vs real, head to head
StatisticallyTS NFNF gtgt Avg TS gt Avg
29Discussion
- Are incentives for virtual currency strong
enough? - Yes (to a degree)
- Conjecture Enough to get what people already
know not enough to motivate independent research - Reduced incentive for information discovery
possibly balanced by better interpersonal
weighting - Statistical validations show HSX, FX, NF are
reliable sources for forecasts - HSX predictions gt expert predictions
- Combining sources can help
30A Problem w/ Virtual CurrencyPrinting Money
Alice1000
Betty1000
Carol1000
31A Problem w/ Virtual CurrencyPrinting Money
Alice5000
Betty1000
Carol1000
32YootlesA Social Currency
Alice0
Betty0
Carol0
33YootlesA Social Currency
I owe you 5
Alice-5
Betty0
Carol5
34YootlesA Social Currency
I owe you 5
credit 5
credit 10
Alice-5
Betty0
Carol5
35YootlesA Social Currency
I owe you 5
I owe you 5
credit 5
credit 10
Alice-5
Betty0
Carol5
36YootlesA Social Currency
I owe you 5
I owe you 5
credit 5
credit 10
Alice3995
Betty0
Carol5
37YootlesA Social Currency
- For tracking gratitude among friends
- A yootle says thanks, I owe you one
38Combinatorial Betting
39Combinatorics ExampleMarch Madness
40Combinatorics ExampleMarch Madness
- Typical todayNon-combinatorial
- Team wins Rnd 1
- Team wins Tourney
- A few other props
- Everything explicit(By def, small )
- Every bet indep Ignores logical probabilistic
relationships
- Combinatorial
- Any property
- Team wins Rnd kDuke gt UNC,NCSTACC wins 5
games - 2263 possible props(implicitly defined)
- 1 Bet effects related bets correctlye.g., to
enforce logical constraints
41ExpressivenessGetting Information
- Things you can say today
- (63 chance that) Obama wins
- GOP wins Texas
- YHOO stock gt 30 Dec 2007
- Duke wins NCAA tourney
- Things you cant say (very well) today
- Oil down, DOW up, Obama wins
- Obama wins election, if he wins OH FL
- YHOO btw 25.8 32.5 Dec 2007
- 1 seeds in NCAA tourney win more than 2 seeds
42ExpressivenessProcessing Information
- Independent markets today
- Horse race win, place, show pools
- Stock options at different strike prices
- Every game/proposition in NCAA tourney
- Almost everything Stocks, wagers, intrade, ...
- Information flow (inference) left up to traders
- Better Let traders focus on predicting whatever
they want, however they want Mechanism takes
care of logical/probabilistic inference - Another advantage Smarter budgeting
43Market CombinatoricsPermutations
- A gt B gt C .1
- A gt C gt B .2
- B gt A gt C .1
- B gt C gt A .3
- C gt A gt B .1
- C gt B gt A .2
44Market CombinatoricsPermutations
- D gt A gt B gt C .01
- D gt A gt C gt B .02
- D gt B gt A gt C .01
- A gt D gt B gt C .01
- A gt D gt C gt B .02
- B gt D gt A gt C .05
- A gt B gt D gt C .01
- A gt C gt D gt B .2
- B gt A gt D gt C .01
- A gt B gt C gt D .01
- A gt C gt B gt D .02
- B gt A gt C gt D .01
- D gt B gt C gt A .05
- D gt C gt A gt B .1
- D gt C gt B gt A .2
- B gt D gt C gt A .03
- C gt D gt A gt B .1
- C gt D gt B gt A .02
- B gt C gt D gt A .03
- C gt A gt D gt B .01
- C gt B gt D gt A .02
- B gt C gt D gt A .03
- C gt A gt D gt B .01
- C gt B gt D gt A .02
45Bidding Languages
- Traders want to bet on properties of orderings,
not explicitly on orderings more natural, more
feasible - A will win A will show
- A will finish in 4-7 A,C,E will finish in
top 10 - A will beat B A,D will both beat B,C
- Buy 6 units of 1 if AgtB at price 0.4
- Supported to a limited extent at racetrack today,
but each in different betting pools - Want centralized auctioneer to improve liquidity
information aggregation
46Example
- A three-way match
- Buy 1 of 1 if AgtB for 0.7
- Buy 1 of 1 if BgtC for 0.7
- Buy 1 of 1 if CgtA for 0.7
B
A
C
47Pair Betting
- All bets are of the form A will beat B
- Cycle with sum of prices gt k-1 gt Match(Find
best cycle Polytime) - Match /gt Cycle with sum of prices gt k-1
- Theorem The Matching Problem for Pair Betting is
NP-hard (reduce from min feedback arc set)
48Automated Market Makers
Thanks Yiling Chen
- A market maker (a.k.a. bookmaker) is a firm or
person who is almost always willing to accept
both buy and sell orders at some prices - Why an institutional market maker? Liquidity!
- Without market makers, the more expressive the
betting mechanism is the less liquid the market
is (few exact matches) - Illiquidity discourages trading Chicken and egg
- Subsidizes information gathering and aggregation
Circumvents no-trade theorems - Market makers, unlike auctioneers, bear risk.
Thus, we desire mechanisms that can bound the
loss of market makers - Market scoring rules Hanson 2002, 2003, 2006
- Dynamic pari-mutuel market Pennock 2004
49Overview Complexity Results
Permutations Permutations Permutations Boolean Boolean Boolean Taxonomy Taxonomy Taxonomy
General Pair Subset General 2-clause Restrict Tourney General Tree
Call Market NP-hard EC07 NP-hard EC07 Poly EC07 NP-hard DSS05 co-NP-complete DSS05 ? ? ?
Market Maker (LMSR) P-hard EC08 P-hard EC08 P-hard EC08 P-hard EC08 Approx STOC08 P-hard EC08 Poly STOC08 P-hard XYZ09 Poly XYZ09
50- March Madness bet constructor
- Bet on any team to win any game
- Duke wins in Final 4
- Bet exotics
- Duke advances further than UNC
- ACC teams win at least 5
- A 1-seed will lose in 1st round
51New Prediction Game Yoopick An Application on
Facebook
52Catalysts
- Markets have long history of predictive accuracy
why catching on now as tool? - No press is bad press Policy Analysis Market
(terror futures) - Surowiecki's Wisdom of Crowds
- Companies
- Google, Microsoft, Yahoo! CrowdIQ, HSX,
InklingMarkets, NewsFutures - Press BusinessWeek, CBS News, Economist,
NYTimes, Time, WSJ, ...http//us.newsfutures.com/
home/articles.html
53CFTC Role
- MayDay 2008 CFTC asks for help
- Q What to do with prediction markets?
- Yahoo!, Google entered suggestions
- Right now, the biggest prediction markets are
overseas, academic (1), or just for fun - CFTC may clarify, drive innovationOr not
54Conclusion
- Prediction Marketshammer market, nail
prediction - Great empirical successes
- Momentum in academia and industry
- Fascinating (algorithmic) mechanism design
questions, including combinatorial betting - Points-paid peers produce prettygood predictions