Learning Prospective Robot Behavior

1
Learning Prospective Robot Behavior

• Shichao Ou and Roderic Grupen
• Laboratory for Perceptual Robotics
• University of Massachusetts Amherst

2
A Developmental Approach

• Infant Learning
• In stages
• Maturation processes
• Parents provide constrained learning contexts
• Protect
• Easy?Complex
• Motion mobile for newborns
• Use brightly colored, easy to pick up objects
• Use building blocks
• Association of words and objects

3
Application in Robotics

• Framework for Robot Developmental Learning
• Role of teacher setup learning contexts that
  make target concept conspicuous
• Role of robot acquire concepts, generalize to
  new contexts by autonomous exploration, provide
  feedback
• Control Basis
• Robot actions are created using combinations of
  lts,?,tgt
• Establish stages of learning by time-varying
  constraints on resources
• Easy ? Complex

4
Example

• Learning to Reach for Objects
• Stage 1 SearchTrack
• Focus attention using single brightly colored
  object (s)
• Limit DOF (t) to use head ONLY
• Stage 2 ReachGrab
• Limit DOF (t) to use one arm ONLY
• Stage 3 Handedness, Scale-Sensitive

Hart et. al, 2008
5
Prospective Learning

• Infant adapts to new situations by prospectively
  look ahead and predict failure and then learn a
  repair strategy

6
Robot Prospective Learning with Human Guidance
7
A 2D Navigation Domain Problem

• 30x30 map
• 6 doors, randomly closed
• 6 buttons
• 1 start and 1 goal
• 3-bit door sensor on robot

8
Flat Learning Results

• Flat Q-Learning
• 5-bit state
• (x,y, door-bit1, door-bit2, door-bit3)
• 4 actions
• up, down, left, right
• Reward
• 1 for reaching the goal
• -0.01 for every step taken
• Learning parameter
• a0.1, ?1.0, e0.1
• Learned solutions after 30,000 episodes

9
Prospective Learning

• Stage 1
• All doors open
• Constrain resources to use only (x,y) sensors
• Allow agent learn a policy from start to goal

10
Prospective Learning

• Stage 2
• Close 1 door
• Robot learns the cause of the failure
• Robot back tracks and finds an earlier indicator
  of this cause

11
Prospective Learning

• Stage 2
• Close 1 door
• Robot learns the cause of the failure
• Robot back tracks and finds an earlier indicator
  of this cause
• Create a sub-task
• Learn a new policy to sub-task

12
Prospective Learning

• Stage 2
• Close 1 door
• Robot learns the cause of the failure
• Robot back tracks and finds an earlier indicator
  of this cause
• Create a sub-task
• Learn a new policy to sub-task
• Resume original policy

13
Prospective Learning Results
Learned solutions lt 2000 episodes
14
Humanoid Robot Manipulation Domain

• Benefits of Prospective Learning
• Adapt to new contexts by maintaining majority of
  the existing policy
• Automatically generates sub-goals
• Sub-task can be learned in a completely different
  state space.
• Supports interactive learning

15
Conclusion

• A developmental view to robot learning
• A framework enables interactive incremental
  learning in stages
• Extension to the control basis learning framework
  using the idea of prospective learning