Language Games and Other Meaningful Pursuits

1
Language Games andOther Meaningful Pursuits

• Michael L. Littman
• mlittman_at_research.att.com
• ATT Labs?Research
• (sort of)

2
Motivation

• Software that can understanding language.
• Question answering
• Constructing databases from text
• Natural-language interaction
• Automatic summarization/briefing
• Semantics tricky word meanings informal.

3
What is Meaning?

• I dont really like bananas, but have long
  respected their humorous potential.
• How capture this sentence?
• Predicate-argument structure?
• Bag of words plus associated terms?
• Evaluate by use.
• Might need multiple reps. or none at all.
• Can we make incremental progress?

4
Language Games

• Like other games
• Evaluation process clean.
• Challenging (and fun!) for people.
• Unlike logical games
• Meaning matters!
• No closed world assumption messy.
• Learning necessary... moving target.
• Machine performance far from humans.

5
Word Games

• Super-human performance common
• Scrabble Maven, near-perfect (Sheppard 02)
• Boggle millisecond solutions (Boyan 98)
• Hangman (Littman 00)
• 99.97 9-letter words under 5 guesses
• 1.35 misses on average

6
Language Game Examples

• Crossword puzzles Acrostics
• Cryptic crosswords Wheel of Fortune
• Trivial Pursuit Tribond
• College Bowl Taboo
• Who Wants to Be a Scattergories Millionaire
• Weakest Link Dictionary
• Synonyms/Analogies

7

• Word meaning matters
• Pages semantically relevant to query
• Multiple methods of determining relevance
• query words near each other on page
• query words appear anywhere on page
• Successful theme combine
• precision high, coverage low
• precision low, coverage high

8
Combination Rule

• Module set coverage ci and precision pi
• Let Rj be the modules selecting response j.
• Pick argmaxj Pi in Rj pi Pi in URj (1-pi)
• If modules have independent precision
• Expected accuracy beats maxi ci pi.

9
Crossword Puzzles (Shortz 98)

• Thesaurus Cut off _ _ _ _ _ _ _ _
• Puns Wordplay Monks head? _ _ _ _ _
• Arts Literature Foundation
• Trilogy author _ _ _ _ _ _
• Popular Culture Pal of Pooh _ _ _ _ _ _
• Encyclopedic Mountain known
• locally as Chomolungma _ _ _ _ _ _ _
• Crosswordese Kind of coal or coat _ _ _

I S O L A T E D
A B B O T
A S I MO V
T I G G E R
E V E R E S T
P E A
10
PROVERB System Design

• Candidate generation (Keim et al. 99)
• Like information retrieval clue implies target
• Variety of approaches used simultaneously
• Merging
• Like meta search engine create master list
• Grid filling (Shazeer et al. 99)
• Like constraint satisfaction fit answer to grid
• Solved via probabilistic optimization

11
Modules ClueDB

• Collection of published clues.
• Nymph pursuer SATYR Bugs pursuer ELMER
• Nymph chaser SATYR Place for an ace SLEEVE
• Highball ingredient RYE Highball ingredient
  ICE
• exact Highball ingredient RYE
• partial Ace place? SLEEVE
• TransModule Bugs chaser ELMER

X chaser? X pursuer
12
Modules Special Purpose

• Database modules Transform clue to DB query.
• imdb Warner of Hollywood OLAND
• wordnet Fruitless ARID
• Syntactic Variations of fill-in-the-blanks.
• blanks_movies Heavens ____! ABOVE
• Web search Not used in experimental system.
• google The Way To Natural Beauty author, 1980
  TIEGS

13
Modules Backstops

• Word lists Ignore clue, return all words.
• wordList 10,000 words, perhaps NOVELETTE
• Implicit modules Probability distributions over
  all strings of words (e.g., bigram).
• segmenter 1934 Hall and Nordhoff adventure
  novel PITCAIRNISLAND

14
Grid Filling
15
Grid Filling
16
Grid Filling
17
Grid Filling
18
Grid Filling
19
Grid Filling
20
Grid Filling
21
Grid Filling
22
Grid Filling
23
Grid Filling
24
Grid Filling
25
Final 88 words, 97 letters
26
PROVERB Results

• Test collection
• 95 words, 98 letters, 46 puzzles
• NYT 89.5 (95.5 MTW, 85.0 TFSS)
• American Crossword Puzzle Tournament
• 1998 190/251, 80 words (vs. 100)
• tricks letter pairs, words in single square
• 1999 147/261, 75 words
• tricks cloonerized spues!

spoonerized clues!
27
Cryptic Crosswords

• Popular in UK
• A clue like this about Japanese religion (6)

SO
HINT
about( HINT, SO)
SHINTO

28
Crossword Maestro (Tunstall-Pedoe)

• Solves this clue
• japanese religion' is the definition. I am not
  sure about the japanese' bit but Shinto' can be
  an answer for religion'.
• a clue like this about' is the subsidiary
  indication. a clue becomes hint', like this'
  becomes so', about' means one lot of letters
  goes inside another. hint' placed inside so'
  is shinto'.
• http//www.genius2000.com/cm.html

29
CM Analysis

• CM knows
• Shinto is a religion (although not Japanese)
• Back-off Japanese religion kind of religion.
• other classes manmade objects, bodily
  activities, natural acts, verbs in base form,
  adjectives
• a clue is hint, like this is so
• about indicates containment (game knowl.)
• Wordnet? http//www.cogsci.princeton.edu/wn/

30
CM Results

• Ads claim 75 on individual clues (perhaps full
  puzzles).
• New York Times puzzles, under 5.
• Very sophisticated
• NL explanations
• broad coverage
• real time

31
Question Answering

• Blend of IR and NLP.
• query, tag type
• find relevant passages
• extract entities, sort by score (combination)

32
Who invented the first computer?

• The stunning shortsightedness and obtusity of
  XEROX's top-level suits has been well anatomized
  in Fumbling The Future How XEROX Invented, Then
  Ignored, the First Personal Computer by Douglas
  K. Smith and Robert C. Alexander (William Morrow
  Co., 1988, ISBN 0-688-09511-9).
• 1876 Alexander Graham Bell invented the
  telephone. 1958 First electronic computers were
  built.
• Steve Jobs and Steve Wozniak were two young men
  who invented the first-ready made computer, the
  Apple.
• A Brief History of Computers and Data
  Communications. 1820 Charles Babbage invented
  the difference engine 1879 First private
  telephone switch board was installed.

33
Trivial Pursuit

• Race around board, answer questions.
• Categories Geography, Entertainment, History,
  Literature, Science, Sports

34
Wigwam

• QA via AQUA (Abney et al. 00)
• back off word match in order helps score.
• When was Amelia Earhart's last flight?
• 1937, 1897 (birth), 1997 (reenactment)
• Move selection via MDP (Littman 00)
• Estimate category accuracy.
• Minimize expected turns to finish.

35
Simulation Results
36
TOEFL Synonyms

• Used in college applications.
• fish
• scale
• angle
• swim
• dredge

37
Synonym Approaches

• Latent Semantic Indexing (Landauer Dumais 97)
• Analyze 30k paragraphs, 300d embedding
• 64 Boulder http//lsa.colorado.edu/
• Pointwise Mutual Information-IR (Turney 01)
• Counts in Altavista (350M) 74 (us 77)
• Thesaurus
• http//Wordsmyth.net 98 prec. 74 cov.
• Combine with PMI-IR 92

38
Verbal Analogies

• Used in college boards (SATs, GREs), and as an
  intelligence test.
• cat meow
• mouse scamper
• bird peck
• dog bark
• horse groom
• lion scratch

39
Preliminary Results

• Undergraduate project at Princeton
• 17 groups
• 229 examples (ETS, web, class)
• 76 test problems from SATs

40
Analogy Results SATs

• Three groups sig. better than chance.

41
Some Successful Techniques

• Individual groups used combinations
• Wordnet for synonyms, antonyms, is-a
• web search with linking words
• x (characterized by OR possesses OR possessed OR
  possess OR characterizes) y
• automatically identify linking words
• word association in definitions

42
Conclusion

• Language games interesting, challenging.
• Progress on solving meaning problems.
• Word order
• Words, as a backoff
• Meaning-preserving transformations
• Probabilistic models
• Combinations!
• Exciting learning and reasoning to be done

43
Challenges

• What year was Dartmouth College founded?
• Dr. Koop graduated from Dartmouth College in 1937
  and received his M.D. degree from Cornell Medical
  College in 1941.
• Although Dartmouth College was established in
  1769 to educate American Indians and