Crowdsourcing and AllPay Auctions

1
Crowdsourcing and All-Pay Auctions

• Milan Vojnovic
• Microsoft Research
• Joint work with Dominic DiPalantino

UC Berkeley, July 13, 2009
2
Examples of Crowdsourcing

• Crowdsourcing soliciting solutions via open
  calls to large-scale communities
• Coined in a Wired article (06)
• Taskcn
• 530,000 solutions posted for 3,100 tasks
• Innocentive
• Over 3 million awarded
• Odesk
• Over 43 million brokered
• Amazons Mechanical Turk
• Over 23,000 tasks

3
Examples of Crowdsourcing (contd)

• Yahoo! Answers
• Lunched Dec 05
• 60M users / 65M answers (as of Dec 06)
• Live QnA
• Lunched Aug 06 / closed May 09
• 3M questions / 750M answers
• Wikipedia

4
Incentives for Contribution

• Incentives
• Monetary
• Non-momentarySocial gratification and
  publicityReputation pointsCertificates and
  levels
• Incentives for both participation and quality

5
Incentives for Contribution (contd)

• Ex. Taskcn

Contest duration Number of submissions Number of
registrants Number of views
Reward range (RMB)
100 RMB ?? 15 (July 09)
6
Incentives for Contribution (contd)

• Ex. Yahoo! Answers

Levels
Points
Source http//en.wikipedia.org/wiki/Yahoo!_Answer
s
7
Questions of Interest

• Understanding of the incentive schemes
• How do contributions relate to offered rewards?
• Design of contests
• How do we best design contests?
• How do we set rewards?
• How do we best suggest contests to players and
  rewards to contest providers?

8
Strategic User Behavior

• From empirical analysis of Taskcn by Yang et al
  (ACM EC 08) (i) users respond to incentives,
  (ii) users learn better strategies
• Suggests a game-theoretic analysis

9
Outline

• Model of Competing Contests
• Equilibrium Analysis
• Player-Specific Skills
• Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion

10
Single Contest Competition
c1
c2
R
c3
c4
contest offeringreward R
players
ci cost per unit effort or quality produced
11
Single Contest Competition (contd)
c1
b1
c2
b2
c3
R
b3
c4
b4
12
All-Pay Auction
v1
b1
v2
b2
v3
b3
v4
b4
Everyone pays their bid
13
Competing Contests
14
Incomplete Information Assumption
Each user u knows
total number of users
his own skill
skills are randomly drawn from F
We assume F is an atomless distribution with
finite support 0,m
15
Assumptions on User Skill
1) Player-specific skill random i.i.d.
across u (ex. contests require similar skills
or skill determined by players
opportunity cost)
2) Contest-specific skill random
i.i.d. across u and j (ex. contests require
diverse skills)
16
Bayes-Nash Equilibrium

• Mixed strategy
• Equilibrium
• Select contest of highest expected profit
  where expectation with respect to beliefs
  about other user skills

prob. of selecting a contest of class j
bid
Contest class set of contests that offer same
reward
17
User Expected Profit

• Expected profit for a contest of class j

prob. of selecting a contest of class j
distribution of user skill conditional on
having selected contest class j
18
Outline

• Model of Competing Contests
• Equilibrium Analysis
• Player-Specific Skills
• Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion

19
Equilibrium Contest Selection
contestclasses
skill levels
20
Threshold Reward

• Only K highest-reward contest classes selected
  with strictly positive probability

number of contests of class k
21
Partitioning over Skill Levels

• User of skill v is of skill level l if

where
22
Contest Selection

• User of skill l, i.e. with skill
  selects a contest of class j with probability

23
Participation Rates

• A contest of class j selected with probability

• Prior-free independent of the distribution F

24
Large-System Limit

• For positive constants
• where K is a finite number of contest classes

25
Skill Levels for Large System

• User of skill v is of skill level l if

where
26
Participation Rates for Large System

• Expected number of participants for a contest of
  class j

• Prior-free independent of the distribution F

27
Contest Selection in Large System

• User of skill l, i.e. with skill
  selects a contest of class j with probability

1/3

• For large systems, what matters is which contests
  are selected for given skill

1/3
1/3
28
Proof Hint for Player-Specific Skills
g1(v)
g2(v)
g3(v)
g4(v)
v
m
0
v1
v2
v3

• Key property equilibrium expected payoffs as
  showed

29
Outline

• Model of Competing Contests
• Equilibrium Analysis
• Player-Specific Skills
• Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion

30
Contest-specific Skills

• Results established only for large-system limit

• Same equilibrium relationship between
  participation and rewards as for player-specific
  skills

31
Proof Hints

• Limit expected payoff For each

• Balancing Whenever

• Asserted relations for follow from above

32
Outline

• Model of Competing Contests
• Equilibrium Analysis
• Player-Specific Skills
• Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion

33
System Optimum Rewards
SYSTEM
maximise
over
subject to

• Set the rewards so as to optimize system welfare

34
Example 1 zero costs(non monetary rewards)

• Assume are increasing strictly concave
  functions. Under player-specific skills, system
  optimum rewards

• for any c gt 0 where ? is unique solution of

• Rewards unique up to a multiplicative constant
  only relative setting of rewards matters

35
Example 1 (contd)

• For large systems

• Assume are increasing strictly concave
  functions. Under player-specific skills, system
  optimum rewards

• for any c gt 0 where ? is unique solution of

36
Example 2 optimum effort

• Consider SYSTEM with

Utility

exerted effort
Cost


cost of giving Rj (budget constraint)
prob. contest attended
37
Outline

• Model of Competing Contests
• Equilibrium Analysis
• Player-Specific Skills
• Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion

38
Taskcn

• Analysis of rewards and participation across
  tasks as observed on Taskcn
• Tasks of diverse categories graphics,
  characters, miscellaneous, super challenge
• We considered tasks posted in 2008

39
Taskcn (contd)
reward
number of views
number of registrants
number of submissions
40
Submissions vs. Reward
Graphics
Characters
Miscellaneous

• Diminishing increase of submissions with reward

linear regression
41
Submissions vs. Rewardfor Subcategory Logos

• Conditional on the rate at which users submit
  solutions

any rate
once a month
every fourth day
every second day

• Conditioning on the more experienced users, the
  better the prediction by the model

model
42
Same for the Subcategory 2-D
any rate
once a month
every fourth day
every second day
model
43
Conclusion

• Crowdsourcing as a system of competing contests
• Equilibrium analysis of competing contests
• Explicit relationship between rewards and
  participations
• Prior-free
• Diminishing increase of participation with reward
• Suggested by the model and data
• Framework for design of crowdsourcing / contests
• Base results for strategic modelling
• Ex. strategic contest providers

44
More Information

• Paper ACM EC 09
• Version with proofs MSR-TR-2009-09
• http//research.microsoft.com/apps/pubs/default.as
  px?id79370