Perception and Perspective in Robotics

1
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and
Artificial Intelligence Laboratory Humanoid
Robotics Group
Toil Example Active Segmentation
Overview
Theft Example Search Activity

• Goal
• To build robots that can interact with novel
  objects and participate in novel activities
• Challenge
• Machine perception can be robust for a specific
  domain such as face detection, but unlike human
  perception it is not currently adaptable in the
  face of change (new objects, changed
  circumstances)
• Approach
• Integrate conventional machine perception and
  machine learning with strategies for
  opportunistic development
• Active perception (sensorimotor toil)
• Interpersonal influences (theft)
• This work is implemented on a humanoid robot
  (Cog, see right). The robot uses the structure of
  familiar activities to learn about novel elements
  within those activities, and tracks known
  elements to learn about the unfamiliar activities
  in which they are used.

Object boundaries are not always easy to detect
visually, so robot Cog sweeps its arm through
ambiguous areas This can cause object motion,
which makes boundaries much easier to find Then
robot can learn to recognize and segment object
without further contact
Robot observes a human searching for objects, and
learns to make a connection between the named
target of the search and the object successfully
found. The robot has no predefined vocabulary or
object set.
Human
Robot
says Find Toma (shows cube) No (shows
car) No (shows bottle) Yes! (shows
cube) Say (shows bottle) Say
says Find Toma (sees cube) No (sees
car) No (sees bottle) Yes (sees
cube) Say Cube (sees bottle) Say Toma
This is a good basis for adaptable object
perception
This work is funded by DARPA under contract
number DABT 63-00-C-10102, and by the Nippon
Telegraph and Telephone Corporation under the
NTT/MIT collaboration agreement
2
Goal To learn how human-level perception is
possible, by trying to build it Challenge Machine
perception can be robust for a specific domain,
but is not adaptable like human
perception Approach Integrate conventional
machine perception and machine learning with
strategies for opportunistic development Active
perception (sensorimotor toil) Interpersonal
influences (theft) Development If a robot is
engaged in a known activity there may be
sufficient constraint to identify novel elements
within that activity. Similarly, if known
elements take part in some unfamiliar activity,
tracking those can help characterize that
activity. Potentially, perceptual development
is an open-ended loop of such discoveries.
Learning a sorting activity Human shows robot
where a collection of disparate objects should
go, based on some common criterion (color). Robot
demonstrates understanding through verbal
descriptions, nods towards target locations.
Kismet What is done on Kismet
Novel Perspective leads to Novel Perception
Learning a search activity Human shows robot
examples of search activity by speaking. Robot
demonstrates understanding by linking name and
object. Learning through a search activity Blah
blah
Cog What is done on Cog
3
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and
Artificial Intelligence Laboratory Humanoid
Robotics Group
An Example Active Segmentation
Overview
Open-ended Development
If the robot is engaged in a known activity there
may be sufficient constraint to identify novel
elements within that activity. Similarly, if
known elements take part in some unfamiliar
activity, tracking those can help characterize
that activity. Potentially, perceptual
development is an open-ended loop of such
discoveries.
Object boundaries are not always easy to detect
visually, so robot Cog sweeps its arm through
ambiguous areas This can cause object motion,
which makes boundaries much easier to find Then
robot can learn to recognize and segment object
without further contact
Goal To learn how human-level perception is
possible, by trying to build it Challenge Machine
perception can be robust for a specific domain,
but is not adaptable like human
perception Approach Integrate conventional
machine perception and machine learning with
strategies for opportunistic development Active
perception (sensorimotor toil) Interpersonal
influences (theft) Experimental
Platform Expressive active vision head Kismet
and upper-torso humanoid robot Cog
Kismet What is done on Kismet
Sorting activity Human shows robot where a
collection of disparate objects should go, based
on some common criterion (color). Robot
demonstrates understanding through verbal
descriptions, nods towards target locations.
Gives opportunity for much development
Cog What is done on Cog
Search activity Human shows robot examples of
search activity by speaking. Robot demonstrates
understanding through verbal descriptions, nods
towards target locations.
4
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and
Artificial Intelligence Laboratory Humanoid
Robotics Group
Active Perception To foo foo foo Active
Segmentation Solve classic problem

• Object boundaries are not always easy to detect
  visually (e.g. yellow car on yellow table)
• Solution robot Cog sweeps through ambiguous area
• Resulting object motion helps segmentation
• Robot can learn to recognize and segment object
  without further contact

• Opportunities abound and cascade
• Robot can perform find the toma style tasks
• Observes search activity
• Then uses structure of search activity to learn
  new properties (object names)
• Searching and sorting

Active Perception Point 1, 2, 3 Motivation Traini
ng examples are currently a necessary condition
for achieving robust machine perception.
Acquiring those examples is properly the role of
perception itself. But a human is typically
needed to collect those examples.
Sorting task Human shows robot where a collection
of disparate objects should go, based on some
common criterion (color). Robot demonstrates
understanding through verbal descriptions, nods
towards target locations.
Search task Human shows robot examples of search
activity by speaking Robot demonstrates
understanding through verbal descriptions, nods
towards target locations.
Active Perception To foo foo foo Active
Segmentation Solve classic problem
5
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and
Artificial Intelligence Laboratory Humanoid
Robotics Group

• Object boundaries are not always easy to detect
  visually (e.g. yellow car on yellow table)
• Solution robot Cog sweeps through ambiguous area
• Resulting object motion helps segmentation
• Robot can learn to recognize and segment object
  without further contact

Goal To understand perception by trying to build
it Approach Extend machine perception to
include opportuistic deve The grist Active
perception Interpersonal influences The
mill Opportunistic development Examples

• Opportunities abound and cascade
• Robot can perform find the toma style tasks
• Observes search activity
• Then uses structure of search activity to learn
  new properties (object names)
• Searching and sorting

6
(No Transcript)
7
Opportunism Standard approach to machine
perception is to develop algorithms which, when
provided with sufficient training data, can learn
to perform some classification or regression
task. Can move one step back and develop
algorithms which, given physical opportunities,
acquire the training data. Need to design system
behavior side-by-side with the perceptual
code. Opportunistic Development Suppose there is
a property P which can normally not be perceived.
But there exists a situation S where it can be.
Then the robot can try to get into situation S,
and observe P, and relate it to other perceptual
variables that are observable
8
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and
Artificial Intelligence Laboratory Humanoid
Robotics Group
1
2
3
(Speech)
9
Training Data
Task Learning Mechanism
Sequencing Model
Instructor
Task Modeling
Understanding perception by trying to build
it Machine perception is very fallible. Robots
(and humans) need not just particular perceptual
competences, but the tools to forge those
competences out of raw physical
experiences. Three important tools for extending
a robots perceptual abilities whose importance
have been recognized individually are related and
brought together. The first is active perception,
where the robot employs motor action to reliably
perceive properties of the world that
it otherwise could not. The second is
development, where experience is used to improve
perception. The third is interpersonal
influences, where the robots percepts are guided
by those of an external agent. Examples are
given for object segmentation, object
recognition, and orientation sensitivity initial
work on action understanding is also described.
Task Grounding
State Grounding
Perceptual System
Demonstrated Task
Perceptual Network
10

• Object boundaries are not always easy to detect
  visually
• Solution Cog sweeps through ambiguous area
• Resulting object motion helps segmentation
• Robot can learn to recognize and segment object
  without further contact

camera image
response for each object
implicated edges found and grouped
11
(No Transcript)
12
(No Transcript)