Title: Bryan Willimon
1Interactive Perception for Cluttered Environments
- Bryan Willimon
- Masters Thesis Defense
2- Visually-guided Manipulation Traditional Approach
Sense
Plan
Act
Manipulation-guided Sensing Interactive
Perception
Act
Sense
Plan
3Previous Related Work on Interactive Perception
- Segmentation through image differencing
Learning about prismatic and revolute joints on
planar rigid objects
D. Katz and O. Brock. Manipulating articulated
objects with interactive perception. ICRA 2008
4Goal of Interactive Perception
Pile of Stuff
Separate Object
Classify
Learn
5Our Approach
- Extraction
- Graph-based Segmentation
- Stereo Matching
- Determining Grasp Point
- Classification
- Color Histogram Labeling
- Skeletonization
- Monitoring Object Interaction
- Labeling Revolute Joints using Motion
6Extraction Process
7Graph-based Segmentation
- Separates the image into regions based on
features of the pixels (e.g., color) - Breaks apart the foreground and background
- Classify background as any pixel that shares the
same color label as a border pixel. - Subtracts background to leave only foreground
8Stereo Matching
- Uses two different cameras from two slightly
different projections to provide a sense of depth - Depth information from foreground only is
considered - Foreground image from previous step is used as a
mask to erase any background information - Object on top of pile minimizes disturbance
9Determining Grasp Point
- Calculate the maximum chamfer distance within
the white area - Use the outline of the white area as the starting
point for the chamfering process - Using chamfer distance instead of centroid
handles concave objects
10Classification
11Color Histogram Labeling
- Use color values (RGB) of the object to create a
3-D histogram - Each histogram is normalized by number of pixels
in object to create a probability distribution - Each histogram is then compared to histograms of
previous objects for a match using histogram
intersection - White area is found by using same technique as in
graph-based segmentation and used as a binary
mask to locate object in image
12Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 1
13Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 3
14Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 5
15Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 7
16Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 9
17Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 10
18Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 11
19Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 13
20Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 15
21Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 17
22Skeletonization
- Use binary mask from previous step to create a
skeleton of the object - Skeleton is a single-pixel wide outline of the
area - Prairie-fire analogy
Iteration 47
23Monitoring Object Interaction
- Use KLT feature points to track movement of the
object as the robot interacts with it - Only concerned with feature points on the object
and disregard all other points - Calculate distance between each feature point
every flength frames (flength5)
24Monitoring Object Interaction (cont.)
- Idea Like features keep a constant
intra-distance, features from different groups
have variable intra-distance - Features were separated into groups by measuring
the intra-distance amount after flength frames - If the intra-distance between two features
changes by less than a threshold, then they are
within the same group - Otherwise, they are within
- different groups
- Separate groups relate to
- separate parts of an object
25Labeling Revolute Joints using Motion
- For each feature group, create an ellipse that
encapsulates all features - Calculate major axis of ellipse using PCA
- End points of major axis correspond to a revolute
joint and the endpoint of the extremity
26Labeling Revolute Joints using Motion (cont.)
- Using the skeleton, locate intersection points
and end points - Intersection points (Red) Rigid or Non-rigid
joints - End points (Green) Interaction points
- Interaction points are locations that the robot
uses to push or poke the object
27Labeling Revolute Joints using Motion (cont.)
- Map estimated revolute joint from major axis of
ellipse to actual joint in skeleton - In the case of groups with size 1, the revolute
joint is labeled to be the closest intersection
point - After multiple interactions from the robot, a
final skeleton is created with revolute joints
labeled (red)
28Experiments
- Items used for experiments
- 3 Logitech Quick-Cam Pro webcams (2 for stereo
system and 1 for classifying) - PUMA 500 robotic arm (or EZ gripper)
- 2 areas were used and located near each other for
easy use of the robotic arm - One was designated as extracted table and the
other as classification table
29Results
Socks and shoes in a hamper EZ gripper
- Toys on the floor PUMA 500
Recycling bin EZ gripper
30Results Toys on
the(cont.)
floor
Final Skeleton used for Classification
31Results Toys on
the(cont.)
floor
Final Skeleton used for Classification
32Results Toys on
the(cont.)
floor
Final Skeleton used for Classification
33Results Toys on
the(cont.)
floor
Final Skeleton used for Classification
34Results Toys on
the(cont.)
floor
Classification Experiment
1
2
3
4
5
6
7
8
35Results Toys on
the(cont.)
floor
Classification Experiment
Rows Query image, Columns Database image
36Results Toys on
the(cont.)
floor
Classification Experiment
Without use of skeleton
37Results Toys on
the(cont.)
floor
Classification Experiment
With use of skeleton
38Results
Recycling(cont.)
bin
39Results
Recycling(cont.)
bin
Without use of skeleton
40Results
Recycling(cont.)
bin
With use of skeleton
41Results Socks
and(cont.)
Shoes
42Results Socks
and(cont.)
Shoes
Only 1 image matched 5, skeleton could not be
used
43Comparison of Related Work
- Comparing objects of the same type to that of
similar work - Pliers from our results compared to shears in
their results
Our approach
Their approach
44How is our work different?
- Our approach handles rigid and non-rigid objects
- Most of the previous work only considers planar
rigid objects - We gather more information with interaction like
a skeleton of the object, color, and movable
joints. - Other works only look to segment the object or
find revolute and prismatic joints - Our approach works with cluttered environments
- Other works only handle a single object instead
of working with multiple items piled together
45Conclusion
- This is a general approach that can be applied to
various scenarios using manipulation-guided
sensing - The results demonstrated that our approach
provided a way to classify rigid and non-rigid
objects and label them for sorting and/or pairing
purposes - This approach builds on and exceeds previous work
in the scope of interactive perception - This approach also provides a way to extract
items out of a cluttered area one at a time with
minimal disturbance - Applications for this project
- Service robots handling household chores
- Map-making robot learning about the environment
while creating a map of the area
46Future Work
- Create a 3-D environment instead of a 2-D
environment - Modify classification area to allow for
interactions from more than 2 directions - Improve the gripper of the robot for more robust
grasping - Enhance classification algorithm and learning
strategy - Use more characteristics to properly label a
wider range of objects
47Questions?