Clustering Appearance for Scene Analysis

1
Clustering Appearance for Scene Analysis

• Sanjeev J. Koppal and Srinivasa Narasimhan

Carnegie Mellon University Sponsors ONR and NSF
2
Factors effecting Scene Appearance
Materials
varying lighting/viewing
Scene Recovery
Camera
Geometry
Acquired Image
Acquired Images
Varying Lighting
3
Scene Recovery
varying lighting/viewing
Materials
Scene Recovery
Geometry
Acquired Images
Varying Lighting
Only works for simple models with few parameters
4
Scene Recovery Known Lighting
varying lighting
Materials
Scene Recovery
Geometry
Acquired Images
Known Lighting
Photometric stereo Goldman et al (2005), Oren
and Nayar (1995)
5
Scene Recovery Known Geometry
varying lighting/viewing
Materials
Scene Recovery
Known Geometry
Acquired Images
Varying Lighting
Inverse Rendering Ramamoorthi and Hanrahan (01)
, Sato et al (97)
6
Scene Recovery Known Materials
varying lighting
Known Materials
Scene Recovery
Geometry
Acquired Images
7
Scene Recovery Orientation Consistency
Example Objects
Lookup
Scene Recovery
Geometry
Acquired Images
Example spheres of known material Hertzmann et
al (2003)
8
Our Idea
varying lighting
Materials
Scene Recovery
Self-Lookup
Geometry
Varying Lighting
Appearance Clusters
We assume orthographic projection of a static
scene with distant lighting.
9
Appearance Profiles
Shared Extrema Locations
Intensity
Multi-Faceted Cylinder
Frame
Same Extrema Locations
Same Surface Normal
Different Extrema Locations
Different Surface Normal
10
Profiles of different materials
Shared Extrema Locations
Intensity
Multi-Faceted Cylinder
Frame
Unshared Extrema Locations
11
Appearance Model

• Linearly Separable Model

Surface Normal
Albedo
Viewing Direction
Light Source Direction
Roughness
Material Terms M terms
Geometry Terms G terms
Pixel Intensity
Assume no cast shadows (for now)
Narasimhan et al (2003), Oren and Nayar (1995),
Klinker et al (1988)
12
Appearance Model
Linearly Separable Model for Appearance Profiles
Same source direction for all scene points
Same fixed viewing for all scene points
Time varying profile
Same Surface Normal
Same Gs
13
Extrema of Linearly Separable Models
Set the derivative of the profile to zero
M
Extrema Solutions lie on a plane defined by
normal M
14
Extrema of Linearly Separable Models

• Set the derivative of the profile to zero

Types of Extrema
Increase
Decrease
Geometry-dependent Extrema
Material-dependent Extrema
15
Light Source Paths
Random Hand-waving
Structured Paths
A Light Dome (Columbia/MERL)
Stanford Gantry
Levoy and Hanrahan (96)
Gu et al (2006)

• Increases geometry-extrema and reduces
  material-extrema.
• No engineered setup
• Interactive

Unknown Geometry
16
Increasing the Number of Geometry-Extrema
Foreshortening in Geometry term
Normal, n
Source, s(t)
scene point
Foreshortening Maximum at Pole
Any circular path creates Maximum Changing
direction creates Minimum
Random waving creates geometry-extrema at every
scene point
17
Decreasing Coincident Material-Extrema
For scene points, 1 and 2
Trivial case Profiles are Identical
Since Gs are randomly generated these events are
unlikely
18
Simulations

• 20000 profiles x 50 normals x 4 BRDF models

Intensity
Frame
Oren-Nayar
Lambertian
Torrance-Sparrow
Oren-Nayar Torrance-Sparrow
19
Using Extrema in Clustering
Intensity
Frame
000000 1 00000000 1 000000000000000 1
Extrema Locations
Objective function is not continuous
Different Number of Extrema
20
Computing a Canonical Profile
Clustering Metric
Intensity
Shifted profile
Frame
Transformed Values
0
Frame
Low Euclidean Distance
Allows comparisons between profiles with
different extrema
Our metric is 1 dot(a,b) where a and b are unit
profiles
Transformation creates a canonical profile
21
Clustering Algorithm
1. Wave a Light Source
2. Detect Extrema Locations
4. Cluster with any ML algorithm using
dot-product metric
3. Apply Transformation
22
Deciding the number of Clusters
k 10
Overcluster, k 20
k 3
k 5
Merge sub-clusters
23
Cast Shadows cause Over-Clustering

• Adding Visibility to the model

Visibility

• Static scene means fixed visibility

New G term
Shadows create valid sub-clusters
24
Clustering Results CURET textures
Ribbed Paper
Sponge
Straw
Slate
Tile
Grass
Wool
Velvet
Steel Wool
Leaf
Sandpaper
Crackers
Styrofoam
Rug
Plaster
Dana et al (1996)
25
Clustering Results CURET textures
26
Clustering Results CURET textures
Clustered Together
27
Clustering Results Curved Surfaces
Piece-wise planar clusters
The clusters quantize the continuous surface
28
Clustering Results Regular Indoor Scenes
Wood
Metal
Tile
29
Clustering Results Regular Indoor Scenes
30
Clustering Results Regular Indoor Scenes
31
Clustering Results Regular Indoor Scenes
Tile
Specularities in plastic
32
Clustering Results Regular Indoor Scenes
33
Clustering Results Regular Indoor Scenes
Textured Cotton Cloth
34
Clustering Results Regular Indoor Scenes
35
Clustering Results WILD Database

• HDR images of an outdoor scene collected over 1
  year
• Complex outdoor illumination effects and
  materials

Narasimhan et al (2002)
36
Clustering Results WILD
Clusters for Outdoor Scene
37
Clustering Results WILD
Using Euclidean metric
Our method
Comparison with clustering original profiles
38
Texture Transfer within a Cluster
Satin Transfer
Velvet Transfer
39
Texture Transfer within a Cluster
The transferred appearance is consistent over time
40
How do clusters help with scene recovery?
Isolate the pixels of a particular cluster

• All these pixels share the same normal
• The number of unknowns is reduced

41
Linear Equations for M and G Terms
Assume G terms are known Linear system in Ms
Assume M terms are known Linear system in Gs
42
Extracting Lambertian Terms
Constrain the first term to be Lambertian

Enable algorithms that only work on Lambertian
scenes
43
Calibrated Photometric Stereo
Mirror Sphere
Recovered Surface Normals
Non-lambertian Cup
44
Uncalibrated Photometric Stereo
Non-lambertian Scene without light probe
Hayakawa (1994), Basri and Jacobs (2001)
45
Recovered Scene Geometry
Structure of Non-lambertian Scene
46
Conclusions

• Derivatives of appearance
• contain geometric information.
• Iso-normal clusters are
• created using extrema.
• Clustering helps with recovery
• of non-lambertian scenes.