Its a 3D World, After All

1
Its a 3D World, After All

• Alyosha Efros
• CMU

2
The sad and miserable life of an object detector
3
People Detection in the Middle Ages
The Empress Theodora with her court. Ravenna,
St. Vitale 6th c.
4
Multiscale processing
5
Many Object Categories
6
Occlusion, Pose
Giotto, The Mourning of Christ, c.1305
7
Perspective
East Doors (1452)
North Doors (1424)
Lorenzo Ghiberti (1378-1455)
8
Perspective gone wild
Piero della Francesca, The Flagellation (c.1469)
9
Real World
10
Objects vs. Scenes
11
Scene Understanding (The Age of Titans)

• Guzman (SEE), 1968
• Hansen Riseman (VISIONS), 1978
• Barrow Tenenbaum 1978

• Brooks (ACRONYM), 1979
• Marr (2 ½ D sketch), 1982
• Ohta Kanade, 1978

12
What Went Wrong?(The Age of Titans)
13
Learning Geometry
Torralba Oliva, 2001
14
Learning Geometry
Hoiem, Efros, Hebert, 2005
Andrew Ng et al, 2006
15
Automatic Photo Pop-up
Geometric Labels
Original Image
16
The World Behind the Image
Automatic Photo Pop-up, SIGGRAPH05
17
Parsing Whole Scene
18
Scene
19
Objects and Scenes

• Biedermans violations (1981)

20
Probability, position (2D)
Torralba et al
21
Size
Torralba et al
22
Position (3D)
Saddeth, Torralba, Freeman, Wilsky, 2006
23
Support, Size
2
?
3
?
1
?
24
Improving Object Detection
25
Improving Object Detection
26
Improving Object Detection
27
Improving Object Detection
28
Hoiem, Efros, Hebert, 2006
29
Qualitative Results
30
Top View
31
Interposition, Depth Layers
32
Discussion Questions

• 1. Can we inject 3D knowledge into
  appearance-based methods?
• 2. Should we build datasets supporting 3D shape?
• 3. Where does segmentation come in?
• 4. Is 3D scene modeling necessary?
• 5. Is explicit 3D (e.g. top-down view) necessary?
• 6. Is depth layer extraction the right problem?
  How to approach it?