Camera%20Culture

1
Computational Photography Advanced Topics
Camera Culture
Ramesh Raskar
Paul Debevec
Jack Tumblin
2
Speaker Jack Tumblin
Associate Professor of Computer Science at
Northwestern Univ. His Look Lab group
pursues research on new methods to capture and
manipulate the appearance of objects and
surroundings, in the hope that hybrid
optical/computer methods may give us new ways to
see, explore, and interact with objects and
people anywhere in the world. During his doctoral
studies at Georgia Tech and post-doc at Cornell,
he investigated tone-mapping methods to depict
high-contrast scenes. His MS in Electrical
Engineering (December 1990) and BSEE (1978), also
from Georgia Tech, bracketed his work as
co-founder of IVEX Corp., (gt45 people as of 1990)
where his flight simulator design work was
granted 5 US Patents. He was an Associate Editor
of ACM Transactions on Graphics (2000-2006), a
member of the SIGGRAPH Papers Committee (2003,
2004), and in 2001 was a Guest Editor of IEEE
Computer Graphics and Applications. http//www.cs
.northwestern.edu/jet
3
Speaker Paul Debevec
Research Associate Professor ,University of
Southern California and the Associate director
of Graphics Research,USC's Institute for
Creative Technologies. Debevec's Ph.D. thesis
(UC Berkeley, 1996) presented Façade, an
image-based modeling and rendering system for
creating photoreal architectural models from
photographs. Pioneer in high dynamic range
photography, he demonstrated new image-based
lighting techniques in his films Rendering with
Natural Light (1998), Fiat Lux (1999), and The
Parthenon (2004) he also led the design of HDR
Shop, the first high dynamic range image editing
program. At USC ICT, Debevec has led the
development of a series of Light Stage devices
used in Spider Man 2 and Superman Returns. He is
the recipient of ACM SIGGRAPH's first Significant
New Researcher Award and a co-author of the 2005
book High Dynamic Range Imaging from Morgan
Kaufmann. http//www.debevec.org
4
Speaker Ramesh Raskar
Associate Professor, MIT Media Lab. Previously
at MERL as a Senior Research Scientist. His
research interests include projector-based
graphics, computational photography and
non-photorealistic rendering. He has published
several articles on imaging and photography
including multi-flash photography for depth edge
detection, image fusion, gradient-domain imaging
and projector-camera systems. His papers have
appeared in SIGGRAPH, EuroGraphics, IEEE
Visualization, CVPR and many other graphics and
vision conferences. He was a course organizer at
Siggraph 2002 through 2005. He was the panel
organizer at the Symposium on Computational
Photography and Video in Cambridge, MA in May
2005 and taught a graduate level class on
Computational Photography at Northeastern
University, Fall 2005. He is a member of the ACM
and IEEE. http//raskar.info http//www.media.mit
.edu/raskar
5
Overview

• Unlocking Photography
• Not about the equipment but about the goal
• Capturing machine readable visual experience
• Goes beyond what you can see through the
  viewfinder
• Push the envelope with seemingly peripheral
  techniques and advances
• Think beyond post-capture image processing
• Computation well before image processing and
  editing
• Learn how to build your own camera-toys
• Emphasis
• Most recent work in graphics/vision (2006 and
  later)
• Research in other fields Applied optics, novel
  sensors, materials
• Review of 50 recent papers and projects
• What we will not cover
• Minimum discussion of graphics/vision papers
  before 2006
• Epsilon photography (improving camera performance
  by bracketing)
• Film Cameras, Novel view rendering (IBR), Color
  issues, Traditional image processing/editing

6
Traditional Photography
Detector
Lens
Pixels
Image
Courtesy Shree Nayar
7
Traditional Photography
Detector
Lens
Pixels
Mimics Human Eye for a Single Snapshot Single
View, Single Instant, Fixed Dynamic range and
Depth of field for given Illumination in a
Static world
Image
8
Traditional Photography
Detector
Lens
Pixels
Image
Courtesy Shree Nayar
9
Computational Photography Optics, Sensors and
Computations
GeneralizedSensor
Generalized Optics
Computations
Ray Reconstruction
4D Ray Bender
Upto 4D Ray Sampler
Merged braketed photos, Coded sensing
10
Computational Photography
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
11
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
12
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
Scene 8D Ray Modulator
13
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
Display
Scene 8D Ray Modulator
Recreate 4D Lightfield
14
Computational Photography
Novel Illumination
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
4D Incident Lighting
4D Ray Bender
Ray Reconstruction
Upto 4D Ray Sampler
4D Light Field
Display
Scene 8D Ray Modulator
Recreate 4D Lightfield
15
What is Computational Photography?

• Create photo that could not have been taken by a
  traditional Camera (?)
• Goal Record a richer, multi-layered visual
  experience
• Overcome limitations of todays cameras
• Support better post-capture processing
• Relightable photos, Focus/Depth of field, Fg/Bg,
  Shape boundaries
• Enables new classes of recording the visual
  signal
• Moment Cohen05, Time-lapse, Unwrap mosaics,
  Cut-views
• Synthesize impossible photos
• Wrap-around views Rademacher and Bishop 1998),
  fusion of time-lapsed events Raskar et al 2004,
  motion magnification Liu et al 2005), video
  textures and panoramas Agarwala et al 2005.
• Exploit previously exotic forms of scientific
  imaging
• Coded aperture Veeraraghavan 2007, Levin 2007,
  confocal imaging Levoy 2004, tomography
  Trifonov 2006

16
Computational Photography

• Epsilon Photography
• Low-level vision Pixels
• Multi-photos by perturbing camera parameters
• HDR, panorama,
• Ultimate camera
• Coded Photography
• Single/few snapshot
• Reversible encoding of data
• Additional sensors/optics/illum
• Scene analysis (Consumer software?)
• Essence Photography
• Beyond single view/illum
• Not mimic human eye
• New art form

17
Epsilon Photography

• Dynamic range
• Exposure bracketing Mann-Picard, Debevec
• Wider FoV
• Stitching a panorama
• Depth of field
• Fusion of photos with limited DoF Agrawala04
• Noise
• Flash/no-flash image pairs
• Frame rate
• Triggering multiple cameras Wilburn04

18
Dynamic Range
Short Exposure
Goal High Dynamic Range
Long Exposure
19
Epsilon Photography

• Dynamic range
• Exposure braketing Mann-Picard, Debevec
• Wider FoV
• Stitching a panorama
• Depth of field
• Fusion of photos with limited DoF Agrawala04
• Noise
• Flash/no-flash image pairs Petschnigg04,
  Eisemann04
• Frame rate
• Triggering multiple cameras Wilburn05,
  Shechtman02

20
Computational Photography

• Epsilon Photography
• Low-level Vision Pixels
• Multiphotos by perturbing camera parameters
• HDR, panorama
• Ultimate camera
• Coded Photography
• Mid-Level Cues
• Regions, Edges, Motion, Direct/global
• Single/few snapshot
• Reversible encoding of data
• Additional sensors/optics/illum
• Scene analysis
• Essence Photography
• Not mimic human eye
• Beyond single view/illum
• New artform

21

• 3D
• Stereo of multiple cameras
• Higher dimensional LF
• Light Field Capture
• lenslet array Adelson92, Ng05, 3D lens
  Georgiev05, heterodyne masks Veeraraghavan07
• Boundaries and Regions
• Multi-flash camera with shadows Raskar08
• Fg/bg matting Chuang01,Sun06
• Deblurring
• Engineered PSF
• Motion Flutter shutterRaskar06, Camera Motion
  Levin08
• Defocus Coded aperture Veeraraghavan07,Levin07,
  Wavefront coding Cathey95
• Global vs direct illumination
• High frequency illumination Nayar06
• Glare decomposition Talvala07, Raskar08
• Coded Sensor
• Gradient camera Tumblin05

22
Digital Refocusing using Light Field Camera
125µ square-sided microlenses
Ng et al 2005
23
Multi-flash Camera for Detecting Depth Edges
24

• 3D
• Stereo of multiple cameras
• Higher dimensional LF
• Light Field Capture
• lenslet array Adelson92, Ng05, 3D lens
  Georgiev05, heterodyne masks Veeraraghavan07
• Boundaries and Regions
• Multi-flash camera with shadows Raskar08
• Fg/bg matting Chuang01,Sun06
• Deblurring
• Engineered PSF
• Motion Flutter shutterRaskar06, Camera Motion
  Levin08
• Defocus Coded aperture Veeraraghavan07,Levin07,
  Wavefront coding Cathey95
• Global vs direct illumination
• High frequency illumination Nayar06
• Glare decomposition Talvala07, Raskar08
• Coded Sensor
• Gradient camera Tumblin05

25
Depth Edges
Left
Top
Right
Bottom
Depth Edges
Canny Edges
26

• 3D
• Stereo of multiple cameras
• Higher dimensional LF
• Light Field Capture
• lenslet array Adelson92, Ng05, 3D lens
  Georgiev05, heterodyne masks Veeraraghavan07
• Boundaries and Regions
• Multi-flash camera with shadows Raskar08
• Fg/bg matting Chuang01,Sun06
• Deblurring
• Engineered PSF
• Motion Flutter shutterRaskar06, Camera Motion
  Levin08
• Defocus Coded aperture Veeraraghavan07,Levin07,
  Wavefront coding Cathey95
• Global vs direct illumination
• High frequency illumination Nayar06
• Glare decomposition Talvala07, Raskar08
• Coded Sensor
• Gradient camera Tumblin05

27
Flutter Shutter Camera
Raskar, Agrawal, Tumblin Siggraph2006
LCD opacity switched in coded sequence
28
Coded Exposure
Traditional
Deblurred Image
Deblurred Image
Image of Static Object
29

• 3D
• Stereo of multiple cameras
• Higher dimensional LF
• Light Field Capture
• lenslet array Adelson92, Ng05, 3D lens
  Georgiev05, heterodyne masks Veeraraghavan07
• Boundaries and Regions
• Multi-flash camera with shadows Raskar08
• Fg/bg matting Chuang01,Sun06
• Deblurring
• Engineered PSF
• Motion Flutter shutterRaskar06, Camera Motion
  Levin08
• Defocus Coded aperture Veeraraghavan07,Levin07,
  Wavefront coding Cathey95
• Decomposition Problems
• High frequency illumination, Global/direct
  illumination Nayar06
• Glare decomposition Talvala07, Raskar08
• Coded Sensor
• Gradient camera Tumblin05

30
"Fast Separation of Direct and Global Components
of a Scene using High Frequency Illumination,"
S.K. Nayar, G. Krishnan, M. D. Grossberg, R.
Raskar, ACM Trans. on Graphics (also Proc. of
ACM SIGGRAPH), Jul, 2006.
31
Separating Reflectance Components
withPolarization-Difference Imaging
cross-polarizedsubsurface component
polarization difference(primarily)specular
component
normal image
32
Computational Photography

• Epsilon Photography
• Multiphotos by varying camera parameters
• HDR, panorama
• Ultimate camera (Photo-editor)
• Coded Photography
• Single/few snapshot
• Reversible encoding of data
• Additional sensors/optics/illum
• Scene analysis (Next software?)
• Essence Photography
• High-level understanding
• Not mimic human eye
• Beyond single view/illum
• New artform

33
(No Transcript)
34
Blind Camera
Sascha Pohflepp, U of the Art, Berlin, 2006
35
Capturing the Essence of Visual Experience

• Exploiting online collections
• Photo-tourism Snavely2006
• Scene Completion Hays2007
• Multi-perspective Images
• Multi-linear Perspective Jingyi Yu, McMillan
  2004
• Unwrap Mosaics Rav-Acha et al 2008
• Video texture panoramas Agrawal et al 2005
• Non-photorealistic synthesis
• Motion magnification Liu05
• Image Priors
• Learned features and natural statistics
• Face Swapping Bitouk et al 2008
• Data-driven enhancement of facial attractiveness
  Leyvand et al 2008
• Deblurring Fergus et al 2006, 2008 papers

36
Scene Completion Using Millions of
PhotographsHays and Efros, Siggraph 2007
37
Capturing the Essence of Visual Experience

• Exploiting online collections
• Photo-tourism Snavely2006
• Scene Completion Hays2007
• Multi-perspective Images
• Multi-linear Perspective Jingyi Yu, McMillan
  2004
• Unwrap Mosaics Rav-Acha et al 2008
• Video texture panoramas Agrawal et al 2005
• Non-photorealistic synthesis
• Motion magnification Liu05
• Image Priors
• Learned features and natural statistics
• Face Swapping Bitouk et al 2008
• Data-driven enhancement of facial attractiveness
  Leyvand et al 2008
• Deblurring Fergus et al 2006, 2008 papers

38
Andrew Davidhazy
39
Unwrap Mosaics Video Editing
Rav-Acha et al Siggraph 2008
40
Capturing the Essence of Visual Experience

• Exploiting online collections
• Photo-tourism Snavely2006
• Scene Completion Hays2007
• Multi-perspective Images
• Multi-linear Perspective Jingyi Yu, McMillan
  2004
• Unwrap Mosaics Rav-Acha et al 2008
• Video texture panoramas Agrawal et al 2005
• Non-photorealistic synthesis
• Motion magnification Liu05
• Image Priors
• Learned features and natural statistics
• Face Swapping Bitouk et al 2008
• Data-driven enhancement of facial attractiveness
  Leyvand et al 2008
• Deblurring Fergus et al 2006, 2008 papers

41
Motion Magnification
Liu, Torralba, Freeman, Durand, Adelson
Siggraph 2005
42

• Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson
Siggraph 2005
43
Motion Magnification
Liu, Torralba, Freeman, Durand, Adelson
Siggraph 2005
44
Capturing the Essence of Visual Experience

• Exploiting online collections
• Photo-tourism Snavely2006
• Scene Completion Hays2007
• Multi-perspective Images
• Multi-linear Perspective Jingyi Yu, McMillan
  2004
• Unwrap Mosaics Rav-Acha et al 2008
• Video texture panoramas Agrawal et al 2005
• Non-photorealistic synthesis
• Motion magnification Liu05
• Image Priors
• Learned features and natural statistics
• Face Swapping Bitouk et al 2008
• Data-driven enhancement of facial attractiveness
  Leyvand et al 2008
• Deblurring Fergus et al 2006, 2007-2008 papers

45
Face Swapping

• Find Candidate face in DB and align
• Tune pose, lighting, color and blend
• Keep result with optimized matching cost

Bitouk et al 2008
46
Computational Photography

• Epsilon Photography
• Low-level vision Pixels
• Multi-photos by perturbing camera parameters
• HDR, panorama,
• Ultimate camera
• Coded Photography
• Mid-Level Cues
• Regions, Edges, Motion, Direct/global
• Single/few snapshot
• Reversible encoding of data
• Additional sensors/optics/illum
• Scene analysis
• Essence Photography
• High-level understanding
• Not mimic human eye
• Beyond single view/illum
• New artform

47
Submit your questions ..

• Today
• What makes photography hard?
• What moments you are not able to capture?
• Future
• What do you expect in a camera or photo-software
  you buy in 2020?
• Please submit by break at 330pm
• Panel Discussion at 510pm

48
Siggraph 2006 16 Computational Photography Papers

• Coded Exposure Photography Motion Deblurring
• Raskar et al (MERL)
• Photo Tourism Exploring Photo Collections in 3D
• Snavely et al (Washington)
• AutoCollage
• Rother et al (Microsoft Research Cambridge)
• Photographing Long Scenes With Multi-Viewpoint
  Panoramas
• Agarwala et al (University of Washington)
• Projection Defocus Analysis for Scene Capture and
  Image Display
• Zhang et al (Columbia University)
• Multiview Radial Catadioptric Imaging for Scene
  Capture
• Kuthirummal et al (Columbia University)
• Light Field Microscopy (Project)

• Hybrid Images
• Oliva et al (MIT)
• Drag-and-Drop Pasting
• Jia et al (MSRA)
• Two-scale Tone Management for Photographic Look
• Bae et al (MIT)
• Interactive Local Adjustment of Tonal Values
• Lischinski et al (Tel Aviv)
• Image-Based Material Editing
• Khan et al (Florida)
• Flash Matting
• Sun et al (Microsoft Research Asia)
• Natural Video Matting using Camera Arrays

49
Siggraph 2007 19 Computational Photography Papers

• Image Analysis Enhancement
• Image Deblurring with Blurred/Noisy Image Pairs
• Photo Clip Art
• Scene Completion Using Millions of Photographs
• Image Slicing Stretching
• Soft Scissors An Interactive Tool for Realtime
  High Quality Matting
• Seam Carving for Content-Aware Image Resizing
• Image Vectorization Using Optimized Gradient
  Meshes
• Detail-Preserving Shape Deformation in Image
  Editing
• Light Field High-Dynamic-Range Imaging
• Veiling Glare in High-Dynamic-Range Imaging
• Ldr2Hdr On-the-Fly Reverse Tone Mapping of
  Legacy Video and Photographs
• Appearance Capture Editing
• Multiscale Shape and Detail Enhancement from
  Multi-light Image Collections
• Computational Cameras
• Active Refocusing of Images and Videos
• Multi-Aperture Photography
• Dappled Photography Mask-Enhanced Cameras for
  Heterodyned Light Fields and Coded Aperture
  Refocusing
• Image and Depth from a Conventional Camera with a
  Coded Aperture

50
Siggraph 2008 19 Computational Photography Papers

• Computational Photography Display
• Programmable Aperture Photography Multiplexed
  Light Field Acquisition
• Glare Aware Photography 4D Ray Sampling for
  Reducing Glare Effects of Camera Lenses
• Light-Field Transfer Global Illumination Between
  Real and Synthetic Objects
• Deblurring Dehazing
• Motion Invariant Photography
• Single Image Dehazing
• High-Quality Motion Deblurring From a Single
  Image
• Progressive Inter-scale and intra-scale Non-blind
  Image Deconvolution
• Faces Reflectance
• Data-driven enhancement of facial attractiveness
• Face Swapping Automatic Face Replacement in
  Photographs (Project)
• AppProp All-Pairs Appearance-Space Edit
  Propagation
• Image Collections Video
• Factoring Repeated Content Within and Among
  Images
• Finding Paths through the World's Photos
• Improved Seam Carving for Video Retargeting
  (Project)
• Unwrap Mosaics A new representation for video
  editing (Project)
• Perception Hallucination

51

• Ramesh Raskar and Jack Tumblin
• Book Publishers A K Peters
• Siggraph 2008 booth 20 off
• Booth 821

52
More ..

• Articles
• IEEE Computer,
• August 2006 Special Issue
• Bimber, Nayar, Levoy, Debevec, Cohen/Szeliski
• IEEE CGA,
• March 2007 Special issue
• Durand and Szeliski
• Science News cover story
• April 2007
• Featuring Levoy, Nayar, Georgiev, Debevec
• American Scientist
• February 2008
• Siggraph 2008
• 19 papers
• HDRI, Mon/Tue 830am
• Principles of Appearance Acquisition and
  Representation
• Bilateral Filter course, Fri 830am
• Other courses .. (Citizen Journalism, Wedn
  145pm)
• First International Conf on Comp Photo, April
  2009

53
Class Computational Photography, Advanced Topics
Debevec, Raskar and Tumblin
Module 1 105 minutes 145 A.1 Introduction
and Overview (Raskar, 15 minutes) 200
A.2 Concepts in Computational Photography
(Tumblin, 15 minutes) 215 A.3 Optics
Computable Extensions (Raskar, 30 minutes)
245 A.4 Sensor Innovations (Tumblin, 30
minutes) 315 Q A (15 minutes)
330 Break 15 minutes Module 2 105 minutes
345 B.1 Illumination As Computing (Debevec,
25 minutes) 410 B.2 Scene and Performance
Capture (Debevec, 20 minutes) 430 B.3 Image
Aggregation Sensible Extensions (Tumblin, 20
minutes) 450 B.4 Community and Social Impact
(Raskar, 20 minutes) 510 B.4 Panel
discussion (All, 20 minutes)
Class Page http//ComputationalPhotography.org