Learning in Worlds with Objects

1
Learning in Worlds with Objects

• Leslie Pack Kaelbling
• MIT Artificial Intelligence Laboratory
• With Tim Oates, Natalia Hernandez, Sarah Finney

2
What is an Agent?

• A system that has an ongoing interaction with an
  external environment
• household robot
• factory controller
• web agent
• Mars explorer
• pizza delivery robot

Environment
Observation
Action
3
Agents Must Learn

• Learning is a crucial aspect of intelligent
  behavior
• human programmers lack required knowledge
• agents should work in a variety of environments
• agents should work in changing environments
• What to learn?
• World dynamics What happens when I take a
  particular action?
• Reward What world states are good?

4
Crisis

• Current state-of-the-art learning methods will
  not work in domains with multiple objects
• These are crucial domains for robots of the
  future.

?
5
Representation

• Learning requires some sort of representation of
  states of the world.
• The choice of representation affects
• what information can be represented
• what kinds of generalizations the agent can make

6
Attribute Vector

• State-of-the-art representation for learning

temperature 48.2 pressure 57.9 mB valve1
open valve2 closed time 1048AM backlog
78 volume 32.2 production 45.5
7
Generalization over Attribute Vectors
x
temp 22
time
pressure temp
time close valve
increase temp
add reagent
open valve
8
Complex Everyday Domains
Attribute vector is impossibly big

• book1-on-book2 true
• book2-on-book1 false
• pen-is-yellow true
• pen-is-blue false
• lamp-on true
• lamp-off false
• ink-bottle-level 50
• lamp-in-bottle false
• bottle-on-lamp false
• paper1-color gray
• paper2-color white
• fabric-behind-lamp true
• book2-is-clear false
• book4-is-clear false
• book1-is-clear true
• block1-on-block2 false
• block3-unstable true
• block2-on-table false
• block1-in-front-of-lamp true

9
Generalization over Objects

• If book1 is on book2 and I move book2, then book1
  will move
• If the cup is on the table and I move the table,
  then the cup will move
• If the pen is on the paper and I move the paper,
  then the pen will move
• If the coat is on the chair and I move the chair,
  then the coat will move
• For all objects A and B
• If A is on B and I move B, then A will move

10
Referring to Objects

• Traditional symbolic AI has the problem of
  symbol grounding
• How do I know what object is named by book1?

on(book1,book2)
11
Deictic Expressions

• Deixis is Greek for pointing

ima
koko
watashi-ga motteiru hako
watashi-ga miteiru hako
12
Automatic Generalization

• If I have an object in my hand and I open my
  hand, then the object that was in my hand is now
  on the table
• This is true, no matter what object is in your
  hand.

13
Communicating with Humans

• Natural language communication
• speaks of the world in terms of objects and their
  relationships
• uses deictic expressions
• Our robots of the future will have to be able to
  understand and generate human descriptions of the
  world

14
Long-Term Research Goal

• A robotic system with hand and cameras that can
• learn to achieve tasks efficiently through trial
  and error
• acquire natural language descriptions of the
  objects and their properties through
  conversation with humans

15
Short-Term Research Plan

• Explore deictic, object-based representation for
  learning algorithms
• build simulated hand-eye robot system that
  manipulates blocks (with real physics)
• have simulated robot learn to carry out tasks
  from trial and error
• Demonstrate empirically and theoretically that
  deictic representation is crucial for efficient
  learning

16
First Example Domain

• Unreliable block stacking
• robot is rewarded for making tall piles of blocks
• the taller a pile is, the more likely it is to
  fall over when another block is added
• a pile can be made more stable by building piles
  to its sides
• Once the robot learns to do this task, keep the
  physics of the domain the same, but reward a more
  complex behavior.

17
Learning by Doing

• Having an initial task to perform focuses the
  robots attention on aspects of the environment
• Use extension of Utree learning algorithm to
  select important aspects of the environment
• Generate new deictic expressions dynamically
  the-block-on-top-of(the-block-I-am-looking-at)
• Extend reinforcement learning methods to apply to
  object-based representations

18
Extracting General Rules

• There are too many facts that are true in any
  interesting environment.
• Solving tasks focuses attention on
• particular objects (named with deictic
  expressions)
• particular properties of those objects
• These objects and properties are likely of
  general importance use them as input to
  association-rule learning algorithm to learn
  facts like
• The thing that is on the thing that I am holding
  will probably fall off if I move

19
Enabling Planning

• Given general rules, the agent can think about
  the consequences of its actions and decide what
  to do, rather than learn through trial and error.

20
In Future

• An ambitious research project
• vision algorithms for learning segmentation and
  object recognition
• learning good properties and relations for
  characterizing the domain (concept learning)
• connect with natural language learning for word
  meanings

21
Dont miss any dirt!