Billion Photos as Generic Source of Data

1
Billion Photosas Generic Source of Data
SIGGRAPH Course Next Billion Cameras
Alexei (Alyosha) Efros Carnegie Mellon University
TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAAA
2
Subject-specific Data
Photos of Coliseum
Portraits of Bill Clinton
3
Much of Captured World is generic
4
Generic Data
street scenes
Food plates
pedestrians
faces
5
Is Generic Data useful?

• A motivating example

6
Hays and Efros. Scene Completion Using Millions
of Photographs. SIGGRAPH 2007 and CACM October
2008.
7
(No Transcript)
8
Diffusion Result
9
Efros and Leung result
10
(No Transcript)
11
Scene Matching for Image Completion
12
Scene Completion Result
13
The Algorithm
14
Scene Matching
15
Scene Descriptor
16
Scene Descriptor
Scene Gist Descriptor (Oliva and Torralba 2001)
17
Scene Descriptor

Scene Gist Descriptor (Oliva and Torralba 2001)
18
2 Million Flickr Images
19
200 total
20
Context Matching
21
Graph cut Poisson blending
22
Result Ranking
We assign each of the 200 results a score which
is the sum of
The scene matching distance
The context matching distance (color
texture)
The graph cut cost
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
200 scene matches
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Why does it work?
38
(No Transcript)
39
Nearest neighbors from acollection of 20
thousand images
40
Nearest neighbors from acollection of 2 million
images
41
Unreasonable Effectiveness of Data
Halevy, Norvig, Pereira 2009

• Parts of our world can be explained by elegant
  mathematics
• physics, chemistry, astronomy, etc.
• But much cannot
• psychology, economics, genetics, etc.
• Enter The Data!
• Great advances in several fields
• e.g. speech recognition, machine translation
• Case study Google

42

• A.I. for the postmodern world
• all questions have already been answeredmany
  times, in many ways
• Google is dumb, the intelligence is in the data

43
How about visual data?

• text is simple
• clean, segmented, compact, 1D
• Visual data is much harder
• Noisy, unsegmented, high entropy, 2D/3D
• Quick Overview
• Uses of Data
• Visual content for graphics
• Better image understanding for vision
• The Dangers of Data

44
Photos as Visual Contentfor Computer Graphics
45
Traditional Computer Graphics
46
State of the Art CG

• Amazingly real
• But so sterile, and lifeless

47
The richness of our everyday world
Photo by Svetlana Lazebnik
48
Appearance Transfer
pixels, texture, color distribution, contrast,
lighting, etc.
a few examples
49
Semantic Photo Synthesis EG06
Johnson, Brostow, Shotton, Arandjelovic, Kwatra,
and Cipolla. Eurographics 2006.
50
Photo Clip Art SG07

• Inserting a single object -- still very hard!

• object size, orientation
• scene illumination

Lalonde et al, SIGGRAPH 2007
51
Photo Clip Art SG07

• Use database to find well-fitting object

Lalonde et al, SIGGRAPH 2007
52
Face Swapping SG08
D. Bitouk, N. Kumar, S. Dhillon, P. N. Belhumeur,
and S. K. Nayar, SIGGRAPH08
53
SkyFinder SG09
Tao, Yuan, Sun, SIGGRAPH 2009
54
Webcam Clip Art SG Asia09
Object transfer
illuminant transfer
Lalonde et al, SIGGRAPH Asia 2009
55
CG2Real
CG2Real Improving the Realism of Computer
Generated Images using a Large Collection of
Photographs, Johnson, Dale, Avidan, Pfister,
Freeman, Matusik, Tech. Rep. MIT-CSAIL-TR-2009-034
56
Image Restoration using Online Photo Collections
ICCV09
Dale, Johnson, Sunkavalli, Matusik, Pfister,
ICCV09
57
Exploring Virtual Space IVW08
Sivic, Kaneva, Torralba, Avidan, Freeman,
Internet Vision Workshop, 2008
58
Photos as Visual Knowledgefor Computer Vision
59
Label Transfer
Tags Sky, Water, Beach, Sunny, Time 1pm,
August, 2006, Location Italy, Greece, Hawaii
Photographer Flickrbug21, Traveller2
a few examples
60
80 Million Tiny Images PAMI08
Torralba, Fergus, Freeman, PAMI 2008
61
80 Million Tiny Images PAMI08
Torralba, Fergus, Freeman, PAMI 2008
62
Non-parametric Scene Parsing CVPR09
Liu, Yuen, Torralba, CVPR 2009
63
im2gps CVPR08
Hays Efros, CVPR 2008
64
Priors for Large Photo Collections
Compute Aggregate Statistic
Independent of Scenes, Photographers Cameras
Camera Distortion Free
Recover Camera Properties
Compute Aggregate Statistic
Independent of Scenes Photographers
One Cameras Distortion
Dependent on Camera
Kuthirummal et al, ECCV08
65
The Dangers of Data
66
Bias

• Internet is a tremendous repository of visual
  data (Flickr, YouTube, Picassa, etc)
• But its not random samples of visual world
• Many sources of bias
• Sampling bias
• Photographer bias
• Social bias

67
Flickr Paris
68
Real Paris
69
Real Notre Dame
70
Sampling Bias

• People like to take pictures on vacation

71
Photographer Bias

• People want their pictures to be recognizable
  and/or interesting

vs.
72
Photographer Bias

• People follow photographic conventions

vs.
73
Social Bias
100 Special Moments by Jason Salavon
74
Social Bias
Mildred and Lisa
Source U.S. Social Security Administration
Gallagher et al CVPR 2008
75
Social Bias
Gallagher et al CVPR 2008
Gallagher et al, CVPR 2009
76
Reducing / Changing Bias
Satellitegoogle.com
Street side Google StreetView
Webcams

• Autonomous capture methods can reduce / change
  bias
• But it wont go away completely
• Need care in picking your data to suit your
  problem

77
Conclusions

• There is lots of generic visual data out there,
  and there will be MUCH more!
• Some of it with useful annotations
• We are beginning to figure out how to use it
• But lots of open questions still remain
• Warning all data is biased
• Need to understand and exploit the biases