Human Computation Play a Game to Develop an Ontology

1
Human ComputationPlay a Game to Develop an
Ontology

• Peyman Nasirifard
• peymab-bn dot sin(arcsin(lastname)) _at_
  deri.org

2
Agenda

• Introduction
• CAPTCHA
• Games with a purpose
• ESP game
• Peekaboom
• Verbosity
• Possible game for developing simple ontologies
• Play a game
• Conclusion

3
Introduction

• Human-based computation is a technique when a
  computational process performs its function via
  outsourcing certain steps to humans.

4
Back to History

• Yahoo! and Gmail are not interested to enable a
  bot to create thousands accounts per day for
  sending spam
• They use CAPTCHA to prevent it

plus
5
CAPTCHA

• Stands for Completely Automated Public Turing
  test to tell Computers and Humans Apart
• Luis von Ahn et al. coined the term in 2000
• A Program that can tell whether a user is a
  human or a computer
• Many different techniques

6
Some Examples
7
Dog or Cat?
Human mmm dog Computer mmmmmmmmmmmmmmm
8
Human Computation

• If we use people to break CAPTCHA, we are doing
  human computation
• In some countries, some companies hire people to
  break CAPTCHA and send spam
• Some companies cleverly use humans to break
  CAPTCHA and send spam
• How?

9
Clever spammers
10
Clever Spammers
Free Nude Photos
Type the word in the box if you want to see the
next picture
11
Really?!

• Jan 2004 world without spam by 2006!
• Huge amount of investment
• Bill Gates receives 4 million spams per day

12
Nice Quote

• Luis von Ahn Instead of hiring people and pay
  them to solve our problems, we can design games
  and people will pay us to play our games and
  solve our large-scale problems!

13
The ESP game

• Object of the game type the same word
• Only thing in common is an image
• Players
• Do not know each other (randomly paired)
• Can not communicate
• Advantages
• Two different sources labels the image
• enjoyable
• labels all images on Google image in a short time
• Help to improve English!
• There are many people that play over 20 hours a
  week

14
The ESP game
Player 1
Player 2

• WOMAN
• CAR

• CAR
• GIRL
• TREE

Agree CAR Get points
15
Taboo words

• Taboo words
• More difficult, but more fun

• CAR
• WOMAN

16
Single version of ESP game

• WOMAN
• CAR

• CAR
• GIRL
• TREE

• The engine records everything from previous
  players
• A single player will actually play with another
  player, but not at the same time

17
Cheating and Repetition

• Problem Agreement on cheating
• Lets label all images with dog
• Solution At random, system gets players test
  images to check whether they play honestly or not
• If they do not play honestly, the system will let
  them play, but nothing will be recorded
• For certainty, only labels which at least N pairs
  agreed upon will be stored

18
The Limitations of ESP

• The ESP Game can label images (and consequently
  tell you whats in them), but it cannot
• Find the objects being labelled
• Determine the way in which the object appears
  does the label carrefer to the text car or
  an actual car in the image?

19

• The place of objects in an image
• Such information would be extremely useful for
  computer vision research

man
dog
20

• The Guesser guesses
• Flower
• Petal
• Butterfly

The Revealer clicks on parts of the image and
shows them to the Guesser.
Server Correct, Butterfly
21
Hints
The label car is ambiguous -- this is
car this is also car
The hints help distinguish the manner in which
the label car appears this is the object
car this is the text car
22
Verbosity

• Collect common-sense facts
• Water quenches thirst
• Sky is blue
• Lions eat meat
• We as human know hundreds of millions common
  sense facts
• Computers do not know
• If know, potentially make them more intelligent
  (e.g. search better)

23
Common sense fact samples

• Milk

• It is liquid
• It is white
• it has lactose
• cereal is eaten with it

24
Verbosity
Narrator MILKis typically near cereal is a
liquid
Guesser
25
Verbosity
Guesser is typically near cereal is a
liquid MILK
Narrator MILK
26
Verbosity
Narrator Object Common sense facts about the
object
Guesser
27
Verbosity
Narrator Object
Guesser Common sense facts about the object
Object
28
Templates

• ___ is a kind of ___. Allows for hierarchical
  categorization.
• ___ is used for ___. Provides information about
  the purpose of a word.
• ___ is typically near/in/on ___ (three
  templates). Provide spatial data.
• ___ is the opposite of ___ / ___ is related to
  ___ (two templates). Provide data about basic
  relations between words.
• ___. In the game, this is a wildcard that
  collects related words.

29
Symmetric vs. Asymmetric

• Verbosity is a asymmetric game, whereas ESP game
  is a symmetric game.
• Symmetric games constraint is number of outputs
  per input
• Asymmetric games constraint is number of inputs
  that produces the same output

30
Possible game to build an ontology

• Several game should work together
• Images come from ESP game
• Not always only those images are selected which
  have one object in it
• i.e. car, bike, monitor, mouse, house
• These images are input to next game which tries
  to catch the properties of objects
• car has colour, car has wheels, car has
  manufacture, car has owner, car has building
  year, etc.

31
Possible game to build an ontology

• Cardinality will be caught by templates, as soon
  as we have properties.
• Car has four wheels
• Car has one plaque
• These sentences will be transferred to OWL
  representation using a mediator.
• The more pairs play the game, the more complex
  the ontology will be

32

• Contact me if you are interested to work on it

33

• This page intentionally left blank

34
Guess what!

• It has usually four wheels
• It has usually one seat
• It is kind of vehicle
• It operates with human power
• It operates with batteries
• It has a break system
• It is a kind of chair

35
Answer
36
Conclusion

• Games are enjoyable!
• More than 9 billion Human-hours of solitaire are
  played each year
• We may cleverly using humans to solve large-scale
  problems by designing interesting games
• Many people play word-guessing games to improve
  their English
• Go and play to promote science!

37
References

• 1 Verbosity A Game for Collecting Common-Sense
  Facts, http//www.cs.cmu.edu/biglou/Verbosity.pdf
  
• 2 Peekaboom A Game for Locating Objects in
  Images, http//www.cs.cmu.edu/biglou/Peekaboom.pd
  f
• 3 Labeling Images with a Computer Game,
  http//www.cs.cmu.edu/biglou/ESP.pdf
• 4 Games with a Purpose, http//www.cs.cmu.edu/b
  iglou/ieee-gwap.pdf
• 5 Wikipedia, http//en.wikipedia.org/wiki/Human-
  based_computation
• 6 We'll End Spam Within 2 Years,
  http//www.connectedhomemag.com/Networking/Article
  s/Index.cfm?ArticleID41587http//news.bbc.co.uk/
  2/hi/business/3426367.stm
• 7 CAPTCHA, http//en.wikipedia.org/wiki/Captcha
  , http//www.captcha.net
• 8 ESP game, www.espgame.org
• 9 Peekaboom game, http//www.peekaboom.org/
• 10 Verbosity game, www.peekaboom.org/verbosity/
• 11 Presentation, http//isandtcolloq.gsfc.nasa.g
  ov/fall2006/presentations/Ahn.ppt
• 12 Presentation, www.aladdin.cs.cmu.edu/workshop
  s/lamps05/Slides/Peekaboom.ppt

38

• Game Over
• peymab-bn dot
  sin(arcsin(lastname)) _at_ deri.org