Efficient Exploration with Latent Structure

1
Efficient Exploration with Latent Structure

• Bethany Leffler
• Michael L. Littman, Alex L. Strehl, Thomas Walsh
• Rutgers Laboratory for Real Life Reinforcement
  Learning
• Rutgers, The State University of New Jersey

2
Motivation

• Human movement/interaction with the environment
• Different surfaces, different behavior
• Learning behavior on each surface
• How does generalization help?

3
The Task

• Maintain given speed independent of slope
• Multiple k-armed bandit problems

4
K-Armed Bandit Problem
5
K-Armed Bandit Problem
Reward
1
2
5
-1
4
6
K-Armed Bandit Problem
Reward
1
2
5
-1
4

• Problem
• Exploration vs Exploitation

7
The Experiment

• l number of states 17
• n types of states (unknown) uphill, flat
• k actions 7 possible motor powers
• Reward -(deviation from target speed)2

8
Tested Algorithms

• Naïve
• No clustering
• Go through each action once for each state
• Clustered
• Hierarchical clustering of states
• Go through each action once for each type of
  state

9
Exploitation Result
10
Exploration Time
11
Optimal Policies

• Naïve median elimination
• Ô(l k ln l)
• Clustering median elimination
• Ô(l ln l n k ln n)

12
Summary

• When nltltl, our approach reduces exploration time
• When nl, the our algorithm performs no worse
  than the naïve algorithm
• If the states are improperly grouped, performance
  suffers

13
Future Work

• States
• X
• Y
• Orientation
• Actions
• Forward
• Turn Left
• Turn Right
• 2 different surface types
• Difficult (sand)
• Easy (pavement)
• Actions taken change next state
• Work with Perceptual Science

G
S
14
Questions?