Robot Learning

1
Robot Learning

• Jeremy Wyatt
• School of Computer Science
• University of Birmingham

2
Plan

• Why and when
• What we can do
• Learning how to act
• Learning maps
• Evolutionary Robotics
• How we do it
• Supervised Learning
• Learning from punishments and rewards
• Unsupervised Learning

3
Learning How to Act

• What can we do?
• Reaching
• Road following
• Box pushing
• Wall following
• Pole-balancing
• Stick juggling
• Walking

4
Learning How to Act Reaching

• We can learn from reinforcement or from a teacher
  (supervised learning)
• Reinforcement Learning
• Action Move your arm (a,b,g)
• You received a reward of 2.1
• Supervised Learning
• Action Move your hand to (a,b,g)
• You should have moved to (d,e,q)

a
(x,y,z)
b
g
5
Learning How to Act Driving

• ALVINN learned to drive in 5 minutes
• Learns to copy the human response
• Feedforward multilayer neural network

Steering wheel position
30
32
6
Learning How to Act Driving
7
Learning How to Act Driving

• Distribution of training examples from on the fly
  learning causes problems
• Network doesnt see how to cope with
  misalignments
• Network can forget if it doesnt see a situation
  for a while
• Answer generate new examples from the on the fly
  images

8
Learning How to Act Driving

• Use camera geometry to assess new field of view
• Fill in using information about road structure
• Transform the target steering direction
• Present as a new training example

9
Learning How to Act Driving
10
Learning How to Act

• Obelix
• Learns to push boxes
• Reinforcement Learning

11
What is Reinforcement Learning?

• Learning from punishments and rewards
• Agent moves through world, observing states and
  rewards
• Adapts its behaviour to maximise some function of
  reward

50
3
-1
-1


r9
r1
r5
r4
s9
s5
s4
s1
s2
s3

a9
a5
a4
a2
a3
a1

12
Return Long term performance

• Lets assume our agent acts according to some
  rules, called a policy, p
• The return Rt is a measure of long term reward
  collected after time t
• The expected return for a state-action pair is
  called a Q value Q(s,a)

13
One step Q-learning

• Guess how good state-action pairs are
• Take an action
• Watch the new state and reward
• Update the state-action value

14
Obelix

• Wont converge with a single controller
• Works if you divide it into behaviours
• But

15
Evolutionary Robotics
16
Learning Maps