Quadric Transfer for Immersive Curved Screen Displays

1
Quadric Transfer for Immersive Curved Screen
Displays

• Ramesh Raskar, Jeroen van Bar, Thomas
  Willwacher, Srinivas Rao
• Mitsubishi Electric Research Labs
• Cambridge, MA USA

2
Curved Screen Displays

• Multiple overlapping projectors
• Goal
• Replace single-proj
• Higher resoltn
• Higher brightness
• Sub-pixel auto-alignment
• Parametric solution

Planetarium
Simulators
3
Dome Projection
Spherical Warping

• Edge-Blended
• Display

Sub-Frames
4
(No Transcript)
5
OutlineSeamless Curved Display

• Multi-projector methods
• Simplified Quadric Image Transfer
• Calibration with camera-pair
• Parametric Rendering solution

6
OutlineSeamless Curved Display

• Multi-projector methods
• Simplified Quadric Image Transfer
• Calibration with camera-pair
• Parametric Rendering solution

7
Multi-Projector Displays
Traditional Auto Alignment Auto Alignment Auto Alignment
Non-Parametric Parametric Head-tracked
Planar DataWalls,Barco, Jupiter Surati99 Raskar99, Chen00, Brown02
Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ? ?
Precise config Costly setup Manual alignment
High maintenance
8
Multi-Projector Displays
Manual Auto Alignment Auto Alignment Auto Alignment
Non-Parametric Parametric Head-tracked
Planar DataWalls,Barco, Jupiter Surati99 Raskar99, Chen00, Brown02
Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ? ?
9
Multi-Projector Displays
Manual Auto Alignment Auto Alignment Auto Alignment
Non-Parametric Parametric Head-tracked
Planar DataWalls,Barco, Jupiter Surati99 Raskar99, Chen00, Brown02
Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ? ?
10
Multi-Projector Displays
Manual Auto Alignment Auto Alignment Auto Alignment
Non-Parametric Parametric Head-tracked
Planar DataWalls,Barco, Jupiter Surati99 Raskar99, Chen00, Brown02
Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ? ?
11
Planar Displaywith parametric approach
R Raskar
12
Planar projective transfer homography

• Two images of 3D points on a plane
• Related by a 3x3 matrix

x A3 x 3 x
M
x
A3 x 3
x
x
x
Proj 1
Proj 2
Proj 1
Proj 2
13
Parametric Image Transfer
X
x
x
Planar Homography
Quadric Transfer
14
Parametric Approach

• Calibration
• Lower camera resolution
• Tolerance for pixel localization errors
• Faster calibration
• Rendering
• Efficient well-defined warping
• Avoids look up tables

15
Multi-Projector Displays
Manual Auto Alignment Auto Alignment Auto Alignment
Non-Parametric Parametric Head-tracked
Planar DataWalls,Barco, Jupiter Surati99 Raskar99, Chen00, Brown02
Curved Flight simulators, Domes, SEOS, Barco, Jarvis97, Raskar98, Yang01 ? ?
16
Curved ScreensView for a Sweet-spot
Projector
Sweet spot
17
Calibration for a Sweet-spot
Projector
Camera at Sweet spot
18
Discretized non-parametric approach
Projector Image
p1
p6
c1
c6
Camera Image
Desired Image

19
Pre-Warped Projection

• Discretized Warping Software
• Spitz - PolyDome

20
OutlineSeamless Curved Display

• Multi-projector methods
• Simplified Quadric Image Transfer
• Calibration with camera-pair
• Parametric Rendering solution

21
Quadric classification
Projectively equivalent to sphere
sphere
ellipsoid
paraboloid
hyperboloid
Ruled quadrics
hyperboloids of one sheet
22
Quadrics
4x4 symmetric matrix,
Nine d.o.f
X
Q
9 points in 3D define quadric
23
Quadric Image Transfer
Shashua97
X
If
,
x
x
21 params, 4 more than necessary !
24
Simplified Quadric Image Transfer
Based on..
X
Homography with polar plane
Projected conic
x
x
17 param warp
25
Simplified Quadric Image Transfer
X
17 param warp
x
x
Planar homography 4 corresponding
pixels Quadric transfer 9 corresponding pixels
26
OutlineSeamless Curved Display

• Multi-projector low cost method
• Simplified Quadric Image Transfer
• Calibration with camera-pair
• Parametric Rendering solution

27
Calibration of Quadric Screens
1
Dome Screen
2
4
3
28
Approach

• Calibration
• At each projector i ,
• Project structured pattern
• View with stereo camera
• Finding camera to projector quadric transfer,
• Run-time
• At each projector i,
• Pre-warp input image using

29
CalibrationFinding relationship between camera
and projector
Low-res Camera 640x480 images But each Projector
1024x768
30
Before Blending
31
After Blending
32
Intensity Correction in Overlap
Projector Framebuffers
33
Projector Framebuffers
34
Projector Framebuffer Intensity Weights
35
OutlineSeamless Curved Display

• Multi-projector low cost method
• Simplified Quadric Image Transfer
• Calibration with camera-pair
• Parametric Rendering solution

36
Rendering a 3D Scene

• Steps at each projector
• (Pre-distort vertex 3D location)
• For each triangle T with vertices Mj
• For each vertex M
• Find pixel m via VirtualViewProjection( M )
• Find warped pixel m via quadricTransfer of m
• Replace M with m

37
Vertex Shader for Quadric Transfer in Cg

• vertout main( appin IN, uniform float4x4
  modelViewProj, uniform float4 constColor,
  uniform float3x3 A, uniform float3x3 E,
  uniform float3 e)
• vertout OUT
• float4 m1 float4(IN.position.x, IN.position.y,
  IN.position.z, 1.0f )
• float4 m, mi float3 m2,mp float scale
•  
• m mul( modelViewProj, m1)
• m2.x m.x/m.w m2.y m.y/m.w m2.z 1
• scale mul(m2, mul(E,m2))
• mp mul(A,m2) sqrt(scale)e
• mi.x m.w (mp.x)/(mp.z)
• mi.y m.w (mp.y)/(mp.z)
• mi.zw m.zw
• OUT.position mi 
• OUT.color0 IN.color0 // Use the original
  per-vertex color specified
• return OUT

ParametricWarp
38
3D Terrain Rendering
39
Parametric Rendering Benefits

• Head tracking support
• Update quadric transfer per frame
• Single pass rendering
• Avoid post-rasterized warp
• Efficient rendering
• Better image quality
• Programmable hardware
• Distributed rendering
• Runs unmodified 3D applications

40
Head Tracked Single Pass Rendering
41
Distributed Rendering with Unmodified Application
42
Subpixel Accurate Registration
43
Convex Dome
44
Acknowledgements

• Mitsubishi Electric Research Labs
• Paul Beardsley, Jay Thornton
• Joe Marks
• Mitsubishi Electric, Japan
• Masato Ogata, Hiroyuki Wada
• Masatoshi Kameyama, Ashizaki

45
Seamless Curved Display

• Multi-projector low cost method
• Simplified Quadric Image Transfer
• Complete Parametric calibrendering solution
• Head tracking support, single pass rendering

www.MERL.com/Projects/Projector/
46
Details I Skipped ..

• Photometric Correction Majumder03
• Depth of field is limited
• Estimating camera and projector params
• Internal and External params
• Issue with near-planar 3D points
• Finding pixels weights for blending
• Non-linear optimization
• Rendering
• Warping and Depth buffer issues

47
Advantages

• Parametric warp
• Lower camera resolution
• Tolerance for pixel localization errors
• Faster calibration
• Efficient well-defined warping

48
What is homography ?

j A3 x 3 i
A3 x 3
a1 a2 a3 b1 b2 b3 c1 c2 c3
jx jy 1
ix iy 1

k
j
i

jx (a i) / (c i)
Proj 1
Proj 2

jy (b i) / (c i)
49
Rendering 2D 3D scene
Concave Dome
Convex Dome
50
Projector Mailing List

• majordomo_at_cs.unc.edusubscribe projector
• Projector bibliography

www.raskar.com/Projector/
51
Vertex Shader for Quadric Transfer in Cg

• vertout main( appin IN, uniform float4x4
  modelViewProj, uniform float4 constColor,
  uniform float3x3 A, uniform float3x3 E,
  uniform float3 e)
• vertout OUT
• float4 m1 float4(IN.position.x, IN.position.y,
  IN.position.z, 1.0f )
• float4 m, mi float3 m2,mp float scale
•  
• m mul( modelViewProj, m1)
• m2.x m.x/m.w m2.y m.y/m.w m2.z 1
• scale mul(m2, mul(E,m2))
• mp mul(A,m2) sqrt(scale)e
• mi.x m.w (mp.x)/(mp.z)
• mi.y m.w (mp.y)/(mp.z)
• mi.zw m.zw
• OUT.position mi 
• OUT.color0 IN.color0 // Use the original
  per-vertex color specified
• return OUT

ParametricWarp
52
Planar Displays
Current Multi-Cube System
MERL Projector Planar Mosaic
Feature Comparison
Many hours Time to install Few minutes
20,000/projector cube Cost 5,000 per projector
15 kg/projector cube heavy support Weight 3 kg/projector, no fixed support structure
Many hours Alignment 10 seconds !
Needs regular care Maintenance Very low
Small gaps Image quality Seamless !, Subpixel alignment
Only high end command and control, advertising Market Middle segment Shops, schools, offices, trade shows
53
Offset Sweetspot
Ideal Sweetspot
54
Quadric Transfer

• Convex or concave quadric surfaces
• Minutes rather than seconds due to necessary
  non-linear optimization of quadric transfer
  parameters
• Sub-pixel accuracy
• Current consumer-class graphics hardware allows
  fast rendering

Concave
Convex
55
Non-linear Refinement
Linear Estimation Error 10 pixels
NonLinear Refinement Error 1.0 pixels
56
Traditional Multi-Projector Setup
Well-defined Overlap
No Overlap
Precise configuration Costly setup Manual
alignment High maintenance
57
Related Work

• Conventional Displays
• Manual alignment, expensive infrastructure
• Jupiter,Trimensions, CAVE, Planetaria, Flight
  Simulators
• Planar Screens
• Camera in loop, auto calibration, low cost
• Exploit homography parameters
• Raskar98,Surati99,Chen00,Brown02
• Curved Screens
• Non-parametric solutions
• Jarvis97,Raskar98,Yang01
• Parametric
• ?, Siggraph 2003