CS 6381: Computational Photography

1
CS 638-1 Computational Photography
2
Basic Information

• Instructor Chuck Dyer
• dyer_at_cs.wisc.edu
• 6379 CS
• Office hours M,W 2-3, and by appointment
• Teaching Assistant Yancan Huang
• hyc_at_cs.wisc.edu
• 5388 CS
• Office hours T,W 4-5, and by appointment
• Class webpagehttp//pages.cs.wisc.edu/cs638-1

3
Today

• What is computational photography?
• Course overview

4
The Camera Obscura
It is impossible for me to speak about the
beauty. All painting is dead, consequently,
because this is Life itself. -- Constantin
Huygens on the camera obscura
R. Gemma Frisius (1545) Observing solar eclipse
of 24 January, 1544. Engraving from De radio
astronomico et geometrico liber. Earliest known
illustration of a camera obscura.
5
Film Camera

• Photography invented about 1825 by Joseph Niepce
• 8 hour exposure time!
• William Henry Fox Talbot invents the calotype in
  1834, which pretty much invents the negative
• http//www.digicamhistory.com

6
Digital Camera
Kodak DCS-100 introduced in 1991 with 1.3 MP
7
What is Computational Photography?

• An extension of traditional (digital) photography
  that combines computational techniques from
  computer vision and computer graphics for
  improving image making

Computer Graphics
Computer Vision
CP
Digital Photography
8
Image Making, Not Just Image Taking

• Overcome limitations of traditional photography
• Image enhancement
• Remove blur, re-focus, re-illumination, noise
  reduction
• Capture and combine more light rays from the
  plenoptic function
• New cameras
• Exploit the billions of images on the web
• Why?
• Improve realism and image quality
• Photojournalism
• Art
• Visual modeling of the physical world
• Communication

9
detector
lens
image
Traditional Camera
10
detector
detector
lens
new optics
compute
image
Traditional Camera
Computational Camera
11
Wide Angle Imaging
12
Catadioptric Cameras for 360 Degree Imaging
13
(No Transcript)
14
(No Transcript)
15
Traditional Computer Graphics
projection
simulation
Slide credit A. Efros
16
State of the Art

• Amazingly real
• But so sterile, lifeless, futuristic

Slide credit A. Efros
17
The richness of our everyday world
Pavia, Italy
Slide credit A. Efros
18
Beauty in complexity
Blue Mountains, Australia
Slide credit A. Efros
19
Which parts are hard to model?
Slide credit A. Efros
20
Use Images to Improve Realism People
On the Tube, London
From Final Fantasy
Slide credit A. Efros
21
Improve Realism Faces / Hair
From Final Fantasy
Photo by Joaquin Rosales Gomez
Slide credit A. Efros
22
Improve Realism Urban Scenes
Photo of l LA
Virtual LA (SGI)
Slide credit A. Efros
23
Improve Realism Nature
River Cherwell, Oxford
Slide credit A. Efros
24
The Realism Spectrum
Computational Photography
Computer Graphics
Photography
Realism Manipulation Ease of capture

• easy to create new worlds
• easy to manipulate objects/ viewpoint
• - very hard to look realistic

• instantly realistic
• easy to aquire
• - very hard to manipulate objects/viewpoint

Slide credit A. Efros
25
What is Computational Photography?

• Definition 1 the use of photographic imagery to
  create graphics content
• Definition 2 The use of computational techniques
  to overcome limitations of conventional
  photography

26
Image Enhancement Overcome the Limitations of
Traditional Photography

• Blur, camera shake, noise, damage

27
Limitations of Traditional Photography

• Limited resolution

28
Limitations of Traditional Photography

• Bad color / no color

29
Limitations of Traditional Photography

• Unwanted objects

30
Limitations of Traditional Photography

• Unfortunate expressions

31
Limitations of Traditional Photography

• Limited dynamic range

32
Limitations of Traditional Photography

• Single viewpoint, static 2D picture

33
Limitations of Traditional Photography

• Single depth of focus

34
Course Overview

• Cameras and image formation
• Texture synthesis and image completion
• Image enhancement and restoration
• Removing camera shake and motion blur
• Image completion
• Combining multiple images
• Compositing
• Panoramas, mosaics, collages
• Warping and morphing
• 3D scene reconstruction
• High dynamic range imaging and tone mapping
• Exploiting billions of images on the web
• Light field cameras, coded apertures

35
Data-driven Image Synthesis
Texture synthesis
Image analogies
Efros Leung (1999), Efros Freeman (2001)
Hertzmann et al. (2001)
Image super-resolution
36
Removing Camera Shake and Motion Blur
Fergus et al. (2006)
Levin (2006)
Yuan et al. (2007)
37
Color Image Manipulation
Colorization
Color Harmonization
Levin et al. (2004)
Cohen-Or et al. (2006)
White Balance Adjustment
Hsu et al. (2008)
38
Image completion
Image completion using millions of photographs
Inpainting
Hays and Efros (2007)
Bertalmio et al. (2000)
Image completion with structure propagation
Sun et al. (2005)
39
Image Compositing
Interactive Digital Photomontage
Agarwala et al. (2004)
Face Swapping
Bitouk et al. (2008)
40
Panoramas and Collages
Panorama stitching
AutoCollage
Rother et al. (2006)
Multi-viewpoint panoramas
Agarwala et al. (2006)
41
Warping and Morphing
Face morphing
Schaefer et al. (2006)
Image deformation using moving least squares
42
From 2D to 3D
Modeling and Rendering Architecture from
Photographs
Debevec et al. (1996)
Photo Tourism / Photosynth
Photo Pop-up
Snavely et al. (2006)
Hoiem et al. (2005)
43
Beyond Conventional Cameras
High Dynamic Range Imaging
Gigapixel Images
Kopf et al. (2007)
Light Field Photography
Ng et al. (2005)
44
Image Forensics

• How to detect manipulated images?

45
What will not be covered

• Photoshop
• New camera technologies (except very briefly)
• Video
• Combining photographic imagery with standard
  graphics imagery

46
Course Requirements

• Class Participation about 10
• Read all papers and other readings
• Come to class
• Ask and answer questions
• Talk to me about your project and presentation
• Homework Assignments about 50
• Play with existing apps such as Photoshop and
  Photosynth
• Implement some methods using Matlab
• Final Project about 30
• Implementation, experimentation, report
• Grading is based primarily on effort and
  initiative so try to be creative!
• May be done individually or in pairs
• Project Presentation about 10
• Brief project presentation during last week of
  the course

47
Texture Transfer
48
Tour into the Picture
Horry et al. (1997)
49
Creating Joiners
Flickr Hockneyesque pool
David Hockney
L. Zelnik-Manor and P. Perona, Automating
Joiners, 5th International Symposium on
Non-Photorealistic Animation and Rendering, 2007
50
Face Morphing
51
Playing around with Faces on hotornot.com
B. Davis and S. Lazebnik, Analysis of Human
Attractiveness Using Manifold Kernel Regression,
ICIP 2008
52
Face Beautification
T. Leyvand, D. Cohen-Or, G. Dror and D.
Lischinski, Data-Driven Enhancement of Facial
Attractiveness, SIGGRAPH 2008
53
Hybrid Images

• A. Oliva, A. Torralba, P.G. Schyns, Hybrid
  Images, SIGGRAPH 2006

54
Background Replacement
Sashi Kumar Penta
55
Infinite Panoramas
J. Sivic, B. Kaneva, A. Torralba, S. Avidan, and
W. Freeman,Creating and Exploring a Large
Photorealistic Virtual Space,Internet Vision
Workshop, 2008
56
Creating Unlikely Juxtapositions
57
Creating Unlikely Juxtapositions
58
Creating Unlikely Juxtapositions
Jeff Wall, Flooded Grave
Scott Mutter, Escalator
59
Visual Rhyming
R. Raguram and S. Lazebnik, Computing Iconic
Summaries for General Visual Concepts, Internet
Vision Workshop, 2008  
60
What is Computational Photography?

• Exploiting the billions of online images and
  video in community photo collections
• Facebook has over 4 billion images
• Getty Images has over 3 billion images
• Flickr has over 2 billion images
• YouTube has over 80 million videos
• Over a billion cell phones with cameras
• Mostly unorganized few tagged or labeled
• How to search, index, organize, share,
  manipulate, combine, extract and use image
  content?

61
What do People Photograph?
From a set of 52 million photos with 188 million
tags, and about 3.7 million unique tags.
Tag classification into broad WordNet categories
Flickr Tag Recommendation based on Collective
Knowledge, Borkur Sigurbjörnsson, Roelof van
Zwol, www 2008
62
Image Retrieval is Hard
Content based techniques for image analysis and
retrieval to a text or image exemplar query
63
Internet vision in action

• http//www.like.com/
• http//tineye.com/login
• http//labs.systemone.at/retrievr/
• http//www.polarrose.com/