Embodied language games for autonomous robots

1
Embodied language games for autonomous robots

• Luc Steels
• Sony Computer Science Laboratory Paris
• University of Brussels
• (VUB Artificial Intelligence Lab)

IJCAI, Pasadena July 2009
2
SHRDLU (Terry Winograd - MIT - 1968/1970
3
Shrdlu (Terry Winograd - MIT - 1968/1970
Open-ended grounded language communication
remains elusive
4
Issues

• Natural language is error-prone, open-ended and
  adaptive
• Grounded language requires a whole systems
  approach
• embodiment, perception, motor control,
  cooperative action, conceptualisation, language
  production, parsing, interpretation, adaptation,
  learning
• There is huge variation across languages, not
  only lexicongrammar but also ontologies
• Language and meaning go beyond the individual
• Collective language is shaped by individuals
• Individual system is shaped by collective

5

•  Very long term research is needed to get a
  handle on human-level natural language. 
  Winograd, 2005
• We need a methodology that can help us to
  understand
• Ongoing adaptation
• Autonomous growth in complexity
• Collective self-organisation

6
Lets get inspiration from evolutionary biology
Darwin, Origins of Species, 1859
7
Evolution of eggspots in cichlid fish
Salzburger, Mack, Verheyen Meyer (2005) BMC
Evol. Biol.
8
Methodology of evolutionary biology

• Select phenomenon (egg spots)
• Understand its function (aid in reproduction)
• Understand how it gets built (geneticsdevelopmen
  t)
• Show that it has selective advantage
• Find out when and how it emerged

9
Example of a language system Spanish basic color
terms can be used to draw attention to objects
verde
10
Example of a language system Spanish basic color
terms can be used to draw attention to objects
Example of a language strategy Name basic color
prototypes
11
Variation
Berlin, B. and P. Kay (1969) Basic color terms
Their universality and evolution. University of
California Press, Berkeley Ca
12
Evolution of Old/Middle English color termsfrom
dominant brightness to full color
Evolution of language strategies

• Torht light (e.g. rodortorht, heavenly bright)
  gt ?
• Scir bright, glimmer gt sheer
• Hador clear gt ?
• Fealu pale gt pale
• Hwit shiny (roof, helmet, silver) gt white
• Blaec dark (hair, crow, night) gt black
• Graeg shine (sea, stars, eyes) gt grey
• Read redreflectivity (gold, sword edge) gt red
• Geolo strong light reflection gt yellow
• Blew (bleu) from shine gt blue
• Brun hue reflectivity gt brown

Casson, R. (1997) Color shift evolution of
English color terms from brightness to hue. In
Hardin, C. and L. Maffi (eds.) Color categories
in thought and language. Cambridge University
Press, Cambridge. p. 224-239
13
Transposing the methodology of evolutionary
biology to AI

• Select phenomenon (color, space, time,
  determiners, case, information structure,
  subordination, anaphora)
• Understand its function (role in communication)
• Understand how it gets built (learninginvention)
  
• Show that it has selective advantage
• Find out when and how it emerged

14
Language Game

• Andrea Could I have some more wine?
• Srini grabs the bottle of water
• Andrea No, wine please.
• Srini Red?
• Andrea Yes.
• Srini pours out red wine
• Andrea Thanks.

Wittgenstein, L. (1953) Philosophical
Investigations. Blackwell, London.
15
(No Transcript)
16
Case study 1 Color
Steels, L. and T. Belpaeme (2005) Coordinating
Perceptually Grounded Categories through
Language. Target article. Beh Brain Sci. 28
(4). 469-490. Bleys, J. and L. Steels (2009)
Linguistic Selection of Language Strategies. A
case study for color. ECAL-09 Bleys, J., M.
Loetzsch, M. Spranger, L. Steels (2009) The
Grounded Color Naming Game. Roman-09
17
The Color Naming Game
18
Cone response curves
rods
Luv
Hurvich-Jameson
19
Semiotic cycle
speaker
perception
perception
conceptualisation
interpretation
production
parsing
hearer
utterance
20
Deconstruct Spanish colors 1. Comprehension and
production
Lillo, J., H. Moreira, I. Vitini, J. Martín
(2007) Locating basic Spanish colour categories
in CIE Luv space Identification, lightness
segregation and correspondence with English
equivalents Psicológica (2007), 28, 21-54.
LUV Color space
21
Deconstruct Spanish colors 1. Comprehension and
production
Communicative Success of 2 simulated Spanish
speakers
Baseline experiment
Bleys J. (2009) forthcoming Ph.d.
22
Deconstruct Spanish colors 2. Learning of the
lexicon
0.3
Lateral inhibition successful - increase
used association - decrease competitors
unsuccessful - decrease used association
LUV Color space
23
Deconstruct Spanish colors 2. Learning of the
lexicon
Communicative success of 2 simulated Spanish
speakers
24
Deconstruct Spanish colors 3. Learning ontology
and lexicon
Sample new prototype
Shift prototype to assimilate new sample
0.8
0.3
Create new prototype if existing ones not
distinctive
25
Deconstruct Spanish colors 3. Learning ontology
and lexicon
26
Deconstruct Spanish color strategy 4.
Self-organisation of lexicon and ontology
Sample new prototype
Shift prototype to assimilate new sample
0.7
0.3
Create new prototype if existing ones not
distinctive
27
Deconstruct Spanish color strategy 4.
Self-organisation of lexicon and ontology
5 agents
28
Example evolution ontology and lexicon for 5
agents
After 2500
After 1000
29
Game
Prototypes
Lexicon
30
Semiotic network
31
Why it works
1. We are dealing with a selectionist system with
Heredity, Variation and Selection
2. We are dealing with a self-organising system
because
communicative success influences which choices
are made by individuals
language emergent structure
32
Inspiration from evolutionary biology
Timema
Nosil et al. 2002 Nature
33
Inspiration from evolutionary biology Comparing
two ecotypes of Timoma On Adenostoma and
Ceanothus
predation present
predation absent
Nosil et al. 2002 Nature
34
Color strategy based on huebrightness
After 2500
35
Other color strategy based on brightness
After 1000
After 3500
36
Comparing two strategies
37
Strategy Selection
Brightness-based strategy
Full-color strategy
Word-13
Word-1
Word-14
Word-6
Word-2
Word-11
Word-15
Word-3
Word-7
Word-4
Word-5
Word-8
Word-9
Word-12
Word-10
38
Ongoing competition
Bleys, J. and L. Steels (2009) Linguistic
Selection of Language Strategies. A case study
for color. ECAL-09
39
Cognitive mechanisms
Heredity, variation, and selection
Language strategies
Language systems
Utterances
Communica-tive outcome
Language Evolution by Linguistic Selection
40
Case study 2 Body
Steels, L. and M. Spranger (2008) The Robot in
the Mirror. Connection Science, 20(4)337-358.
Steels, L. and M. Spranger (2008) Can body
language shape body image? ALife XI Ma.
577-584. Steels, L. and M. Spranger (2009) How
experience of the body shapes language about
space. IJCAI-09
41

• Commands
• Stand!
• Right-arm up!
• Posture description
• Thomas sits
• Holk stands
• Positional description of objects
• The clothes lie on the counter
• The bottle stands on the table
• Metaphorical description
• De economie zit in een recessie (Dutch)
• The economy sits in a recession

Lemmens, M. (2002) The semantic network of
Dutch posture verbs. In J. Newman (Ed.) (2002)
The linguistics of sitting, standing and lying.
Amsterdam/Philadelphia John Benjamins.
42
Action Game
Steels, Spranger (2008) Can body language shape
body image? Artificial Life XI. The MIT Press,
Cambridge Ma. pp. 577-584.
43
(No Transcript)
44
Developing body models motor, visual,
proprioception, simulation
Visual Feature Extraction
Normalised Centralised Moments
Hu, M. (1962) Visual Pattern Recognition by
Moment Invariants, IER Trans. Information Theory,
IT-8, pp. 179- 187.
Steels, Spranger (2008) The Robot in the Mirror.
Connection Science, 20(4)337-358.
45
Recording of motor babbling
46
Exaptation after 1000 games
Steels, L. and M. Spranger (2009) How experience
of the body shapes language about space.
IJCAI-09.
47
Case study 3 Space
Steels, L. and M. Loetzsch (2009) Perspective
Alignment in Spatial Language. In Coventry, K.,
J. Bateman and T. Tenbrink (2009) Spatial
Language in Dialogue. Oxford University Press.
Oxford. Loetzsch, M., van Trijp, R. and
Steels, L. (2008) Typological and Computational
Investigations of Spatial Perspective. In
Wachsmuth, I. and Knoblich, G., editor, Modeling
Communication with Robots and Virtual Humans,
LNCS, Springer-Verlag, Berlin.
48
The spatial reference game
The block in front of me. The block facing
me The block North (given the global landmark)
The block to your right. The right block The
block to the right of the box (for you)
49
(No Transcript)
50
?objects
Geometric-transform
?transformed-objects
?landmark
51
?objects
Filter-set-angular-category
?angular-category
?filtered-objects
52
?objects
Filter-set-radial-category
?radial-category
?filtered-objects
53
Get-objects-from-context
?Objects
Filter-set-category
BLOCK
?Blocks
Geometric-transform
HEARER
?Your-blocks
Filter-set-angular-category
RIGHT
Select-object
?Your-right-blocks
?referent
The block to your right
UNIQUE
54
 the X to Y Z  construction
Fluid Construction Grammar
55
Conclusion

• Target open-ended grounded language
• Not only lexicon and grammar but also ontology is
  adaptive and emergent
• Inspiration transpose methodology of
  evolutionary biology
• Deconstruct language systems strategies
• Set up selectionist dynamics
• Case studies
• Color, space, body language
• Many more need to be done!

56
Thanks to the teams!

• University of Brussels (VUB AI Lab)
• Joris Bleys
• Thomas Cederborg
• Joachim De Beule
• Carl Jacobs
• Martin Loetzsch
• Pieter Wellens

• Sony CSL Paris
• Katya Gerasymova
• Vanessa Micelli
• Damien Munch
• Simon Pauw
• Michael Spranger
• Remi van Trijp
• Pieter Wellens

57
finis
58
Case study 2 Scaling up lexicon
GAVAGAI
Quine
Wellens, P., M. Loetzsch, and L. Steels (2008)
Flexible Word Meaning in Embodied Agents.
Connection Science, 20 (2), 173-191.