It

1
Its a 3D World!Toward a Qualitative
Representation of a Scene

• Alyosha Efros
• Carnegie Mellon University

2
The Problem

• Recovering 3D structure from single 2D projection
  
• Infinite number of possible solutions!

from Sinha and Adelson 1993
3
Geometry school disambiguate
J.J. Gibsons Ecological Optics the actively
exploring organism
4
3D Data Capture

• Stereo
• Structure from Motion
• Laser Range Scanning
• Etc.

3D Point Cloud
5
The World Behind the Image
Hoiem, Efros, Hebert, Automatic Photo Pop-up,
SIGGRAPH05
6
Our World is Structured
Abstract World
Image Credit (left) F. Cunin and M.J. Sailor,
UCSD
7
Pattern Recognition school learn!

• recognize full scenes
• Scene gist TorralbaOliva,2001
• 32x32 images Torralba et al,2007
• (3232)2563 huge space!
• recognize all objects in scene
• scan (rectangular) templates
• recognize each object independently

8
Local Object Detection
True Detection
False Detections
Missed
Missed
True Detections
Local Detector Dalal-Triggs 2005
9
What the Detector Sees
10
2D Context is not enough
Close
Not Close
11
Geometrically Coherent Image Interpretation

• Derek Hoiem, Alyosha Efros, Martial Heber

12
Recognizing (qualitative) Geometry

• Goal learn labeling of image into 7 Geometric
  Classes
• Support (ground)
• Vertical
• Planar facing Left (?), Center ( ), Right (?)
• Non-planar Solid (X), Porous or wiry (O)
• Sky

?
13
Learn from labeled data

• 300 outdoor images from Google Image Search

14
Weak Geometric Cues
15
Hypothesizing Regions

• Naïve Idea 1 segment the image
• Chicken Egg problem
• Naïve Idea 2 multiple segmentations
• Decide later which segments are good

16
Labeling Segments


Using Boosted Decision Tree classifier Trained on
labeled data
17
Image Labeling
Labeled Segmentations

Learned from training images
Labeled Pixels
18
No Hard Decisions
Support
Vertical
Sky
V-Center
V-Right
V-Porous
V-Solid
V-Left
19
Labeling Results
20
How robust is it?
21
Shadow/Reflection Failures
Input image
Ground Truth
Our Result
22
Catastrophic Failures
Input image
Ground Truth
Our Result
23
Average Accuracy
Main Class 88.1 Subclasses 61.5
24
Understanding a Scene

• Biedermans Relations among Objects in a
  Well-Formed Scene (1981)

Support Size Position

• Interposition
• Likelihood of Appearance

Object Support
Object Size
25
Object Support
26
Object Size ? Camera Viewpoint
Object Position/Sizes
Viewpoint
27
More Chickens, More Eggs
28
Input to Our Algorithm
Surface Estimates
Viewpoint Prior
Object Detection
Local Car Detector
Local Ped Detector
Surfaces Hoiem-Efros-Hebert 2005
Local Detector Dalal-Triggs 2005
29
Scene Parts Are All Interconnected
Objects
3D Surfaces
Viewpoint
30
Helping Object Detection
Image
P(object)
31
Helping Object Detection
Image
P(surfaces)
P(viewpoint)
P(object surfaces, viewpoint)
P(object)
32
Qualitative Results
Car TP / FP Ped TP / FP
Initial 2 TP / 3 FP
Final 7 TP / 4 FP
Local Detector from Murphy-Torralba-Freeman 2003
33
Qualitative Results
Car TP / FP Ped TP / FP
Initial 1 TP / 14 FP
Final 3 TP / 5 FP
Local Detector from Murphy-Torralba-Freeman 2003
34
Qualitative Results
Car TP / FP Ped TP / FP
Initial 1 TP / 23 FP
Final 0 TP / 10 FP
Local Detector from Murphy-Torralba-Freeman 2003
35
Qualitative Results
Car TP / FP Ped TP / FP
Initial 0 TP / 6 FP
Final 4 TP / 3 FP
Local Detector from Murphy-Torralba-Freeman 2003
36
Top View
Ped
Ped
Car
37
What next?
38
The Challenges
39
Challenges
40
Summary

• 3D is important
• Size, Support, Occlusion, etc. all inherently
  3D phenomena!
• But exact 3D not needed
• Coarse surface layout
• Coarse depth layering
• Coarse viewpoint inference
• Human seem to have only qualitative 3D

41
Automatic Photo Pop-up
Geometric Labels
Original Image
42
More Pop-ups
43
The Music Video
44
Thank you
Questions?