Georgioss Visions interactive learning representations

1
Georgioss Visions(interactive learning ?
representations)
HHMM

• MIT CSAIL

2
Ed Wood(Characterized as the worst film maker
ever)

• "Home? I have no home Hunted,despised, Living
  like an animal! The jungle is my home. But I will
  show the world that I can be its master! I will
  perfect my own race of people. A race of atomic
  supermen which will conquer the world!"

3
Why?

• Learning from delayed reward is hopeless (in my
  opinion)
• Supervised learning is impractical
• Humans and animals live in societies
• Need something above RL and below supervised
  learning

4
Possible Titles

• Social learning
• Interactive learning
• Learning to communicate
• Classroom learning
• Competitive learning

• Do what I mean not what I say
• What do you mean?
• Lets talk
• Robot apprentices
• Searching for the right representations

5
Final Product
PHYSICAL ENVIRONMENT
Observations, Actions,Rewards, State modification
Eriks representation
Georgioss representation
Pavlovs representation
6
Obstacles

• A mathematical framework for interactive learning
  (reward shaping?)
• What are objects (sensory, motor sequences ?)
• How do they relate to each other. What are the
  representations (atomic, propositional,
  first-order?)

7
Example Systems

• A robot that learns to navigate by interaction
  with a human trainer
• A personalized web agent(active information
  extraction)
• Personal assistants (office)

8
Tools Concepts

• H-POMDPS?
• What is missing?
• Dynamic abstractions (structure learning)
• Teleological abstractions
• Relational structure
• Factorization (hierarchical reuse)
• Multiagency /concurrency

9
Grounded Projects

• Other H-POMDP applications
• Model reduction in POMDPs with macros
• Structure learning of H-POMDPs
• Theoretical localization results in grid-worlds
  with structure
• Mathematical framework for interactive learning
• Efficient algorithms for learning stochastic
  models

10
Other H-POMDP Applications

• Passive hierarchical HMM applications
• Policy recognition (AMM) (Hung Bui)
• Video Structure discovery (HHMM) (Lexing Xie)
• Human activity recognition (Nuria Oliver)
• Emotion Recognition (multi level HMM) (Ira
  Cohen)
• Natural English text cursive hand-writing
  (HHMM) (Fine)
• Information extraction (HHMM) (skounakis)
• Active recognition/learning
• Active object detection/recognition (RL) (Lucas
  paletta)
• Selective perception policies for guiding sensing
  (layered HMM ) (Nuria Oliver, Eric Horvitz)
• Active learning of HMMs (Tobias Scheffer)
• What can we do (active learning?) (active
  recognitionPOMDP planning?)
• Recognition of office activity / Active
  recognition of office activity / Active learning
  of model parameters

11
POMDPs Macro-Actions

• A model based RL over a dynamic grid abstraction
  in belief space with macro-actions (NIPS 2003)
• Consider only needed part of belief space
• Learn faster than just using primitive actions
• Ability to do information gathering
• Whats next?
• A new minimized POMDP other than than the belief
  state representation (PSRs? Non-linear
  dimensionality reductions? Smaller HMMs?)
• Other domains

12
Structure Learning

• Natural Language approaches
• Sequitor (Nevill-Manning)
• Unsupervised Language acquisition (Carl G. de
  Marcken)
• Structure learning in graphical models
• Discovering hidden state (X. Boyen)
• From Data Mining
• Bursty and Hierarchical structure in streams (Jon
  Kleinberg)

13
Localizing in Flat Grid Worlds is NP-hard

• In flat POMDPs finding localization plans that
  are within a log factor of optimal is NP-Hard
  (Sven Koenig)
• Does the same hold for H-POMDPs?

14
Mathematical Framework for Interactive learning
ENVIRONMENT
State s
T
State s
Reward r
R
O
Policy
Reward r
z
AGENT
Supervisor
Action a
15
Interactive Learning Literature

• Programmable RL agents (David Andre)
• Principle methods for advising RL agents
  (Garrison Cottrell)
• Machine discovery of effective admissible
  heuristics (Armand E. Prieditis)
• Supervised learning combined with an actor-critic
  architecture (Michaels Rosenstein)
• Shaping in RL by changing the physics of the
  problem (Jette Randolv)
• What if the teacher needs to learn too?

16
Efficient Learning Algorithms for Models of
Stochastic Processes

• Parameter learning in graphical models is
  inefficient (structure learning impractical)
• Can we do better?
• Train model where it needs to be trained
• Do informed searching when learning structure

17
Conclusions

• Big results require big ambitions
• To make progress towards AI,We need to make
  learning and planning more interactive
• This will keep me busy for a while