Title: Clustering Appearance for Scene Analysis
1Clustering Appearance for Scene Analysis
- Sanjeev J. Koppal and Srinivasa Narasimhan
Carnegie Mellon University Sponsors ONR and NSF
2Factors effecting Scene Appearance
Materials
varying lighting/viewing
Scene Recovery
Camera
Geometry
Acquired Image
Acquired Images
Varying Lighting
3Scene Recovery
varying lighting/viewing
Materials
Scene Recovery
Geometry
Acquired Images
Varying Lighting
Only works for simple models with few parameters
4Scene Recovery Known Lighting
varying lighting
Materials
Scene Recovery
Geometry
Acquired Images
Known Lighting
Photometric stereo Goldman et al (2005), Oren
and Nayar (1995)
5Scene Recovery Known Geometry
varying lighting/viewing
Materials
Scene Recovery
Known Geometry
Acquired Images
Varying Lighting
Inverse Rendering Ramamoorthi and Hanrahan (01)
, Sato et al (97)
6Scene Recovery Known Materials
varying lighting
Known Materials
Scene Recovery
Geometry
Acquired Images
7Scene Recovery Orientation Consistency
Example Objects
Lookup
Scene Recovery
Geometry
Acquired Images
Example spheres of known material Hertzmann et
al (2003)
8Our Idea
varying lighting
Materials
Scene Recovery
Self-Lookup
Geometry
Varying Lighting
Appearance Clusters
We assume orthographic projection of a static
scene with distant lighting.
9Appearance Profiles
Shared Extrema Locations
Intensity
Multi-Faceted Cylinder
Frame
Same Extrema Locations
Same Surface Normal
Different Extrema Locations
Different Surface Normal
10Profiles of different materials
Shared Extrema Locations
Intensity
Multi-Faceted Cylinder
Frame
Unshared Extrema Locations
11Appearance Model
Surface Normal
Albedo
Viewing Direction
Light Source Direction
Roughness
Material Terms M terms
Geometry Terms G terms
Pixel Intensity
Assume no cast shadows (for now)
Narasimhan et al (2003), Oren and Nayar (1995),
Klinker et al (1988)
12Appearance Model
Linearly Separable Model for Appearance Profiles
Same source direction for all scene points
Same fixed viewing for all scene points
Time varying profile
Same Surface Normal
Same Gs
13Extrema of Linearly Separable Models
Set the derivative of the profile to zero
M
Extrema Solutions lie on a plane defined by
normal M
14Extrema of Linearly Separable Models
- Set the derivative of the profile to zero
Types of Extrema
Increase
Decrease
Geometry-dependent Extrema
Material-dependent Extrema
15Light Source Paths
Random Hand-waving
Structured Paths
A Light Dome (Columbia/MERL)
Stanford Gantry
Levoy and Hanrahan (96)
Gu et al (2006)
- Increases geometry-extrema and reduces
material-extrema. - No engineered setup
- Interactive
Unknown Geometry
16Increasing the Number of Geometry-Extrema
Foreshortening in Geometry term
Normal, n
Source, s(t)
scene point
Foreshortening Maximum at Pole
Any circular path creates Maximum Changing
direction creates Minimum
Random waving creates geometry-extrema at every
scene point
17Decreasing Coincident Material-Extrema
For scene points, 1 and 2
Trivial case Profiles are Identical
Since Gs are randomly generated these events are
unlikely
18Simulations
- 20000 profiles x 50 normals x 4 BRDF models
Intensity
Frame
Oren-Nayar
Lambertian
Torrance-Sparrow
Oren-Nayar Torrance-Sparrow
19Using Extrema in Clustering
Intensity
Frame
000000 1 00000000 1 000000000000000 1
Extrema Locations
Objective function is not continuous
Different Number of Extrema
20Computing a Canonical Profile
Clustering Metric
Intensity
Shifted profile
Frame
Transformed Values
0
Frame
Low Euclidean Distance
Allows comparisons between profiles with
different extrema
Our metric is 1 dot(a,b) where a and b are unit
profiles
Transformation creates a canonical profile
21Clustering Algorithm
1. Wave a Light Source
2. Detect Extrema Locations
4. Cluster with any ML algorithm using
dot-product metric
3. Apply Transformation
22Deciding the number of Clusters
k 10
Overcluster, k 20
k 3
k 5
Merge sub-clusters
23Cast Shadows cause Over-Clustering
- Adding Visibility to the model
Visibility
- Static scene means fixed visibility
New G term
Shadows create valid sub-clusters
24Clustering Results CURET textures
Ribbed Paper
Sponge
Straw
Slate
Tile
Grass
Wool
Velvet
Steel Wool
Leaf
Sandpaper
Crackers
Styrofoam
Rug
Plaster
Dana et al (1996)
25Clustering Results CURET textures
26Clustering Results CURET textures
Clustered Together
27Clustering Results Curved Surfaces
Piece-wise planar clusters
The clusters quantize the continuous surface
28Clustering Results Regular Indoor Scenes
Wood
Metal
Tile
29Clustering Results Regular Indoor Scenes
30Clustering Results Regular Indoor Scenes
31Clustering Results Regular Indoor Scenes
Tile
Specularities in plastic
32Clustering Results Regular Indoor Scenes
33Clustering Results Regular Indoor Scenes
Textured Cotton Cloth
34Clustering Results Regular Indoor Scenes
35Clustering Results WILD Database
- HDR images of an outdoor scene collected over 1
year - Complex outdoor illumination effects and
materials
Narasimhan et al (2002)
36Clustering Results WILD
Clusters for Outdoor Scene
37Clustering Results WILD
Using Euclidean metric
Our method
Comparison with clustering original profiles
38Texture Transfer within a Cluster
Satin Transfer
Velvet Transfer
39Texture Transfer within a Cluster
The transferred appearance is consistent over time
40How do clusters help with scene recovery?
Isolate the pixels of a particular cluster
- All these pixels share the same normal
- The number of unknowns is reduced
41Linear Equations for M and G Terms
Assume G terms are known Linear system in Ms
Assume M terms are known Linear system in Gs
42Extracting Lambertian Terms
Constrain the first term to be Lambertian
Enable algorithms that only work on Lambertian
scenes
43Calibrated Photometric Stereo
Mirror Sphere
Recovered Surface Normals
Non-lambertian Cup
44Uncalibrated Photometric Stereo
Non-lambertian Scene without light probe
Hayakawa (1994), Basri and Jacobs (2001)
45Recovered Scene Geometry
Structure of Non-lambertian Scene
46Conclusions
- Derivatives of appearance
- contain geometric information.
-
- Iso-normal clusters are
- created using extrema.
- Clustering helps with recovery
- of non-lambertian scenes.