Interactive Virtual Relighting and Remodelling of Real Scenes

1
Interactive Virtual Relighting and Remodelling of
Real Scenes

• C. Loscos1, MC. Frasson1,2,G. Drettakis1,
• B. Walter1, X. Granier1, P. Poulin2
• (1) iMAGIS - GRAVIR/IMAG - INRIA Rhône-Alpes
• iMAGIS is a joint project of
  CNRS/INRIA/UJF/INPG
• (2) Département d informatique et de recherche
  opérationnelle,
• Université de Montréal

2
Motivation
Interior design
Geometric modification
Changes in lighting
3
Motivation

• Goal interactive system
• simple capture process
• interactive ( 1 sec. per frame)
• modification of lighting
• modification of geometry

4
Motivation

• We have to
• create a simple model of the real scene
• geometry
• approximate reflectance
• represent real global illumination
• develop interactive methods for modifications
• Goal is to be convincing, not highly accurate

5
Previous Work

• Geometric reconstruction
• vision methods Faugeras et al. 97, ...
  (Realise)
• constraint-based systems Debevec et al. 96,
  Poulin et al. 98
• software Photomodeler, etc.
• Reflectance recovery
• e.g., Sato et al. 97, Ward92, Debevec98, Yu et
  al. 98, etc.

6
Previous Work

• Real-time direct shadows
• real point light source State et al. 96
• Common global illumination
• non-interactive
• Nakamae et al. 86, Fournier et al. 93, Jancène
  et al. 95, Debevec 98, Yu et al. 98, Yu et al.
  99
• interactive
• Drettakis et al. 97, Loscos et al. 98

7
Algorithm Overview

• Input
• Pre-process
• Interactive modification

8
Algorithm Overview - Assumptions

• Single viewpoint
• Diffuse assumption
• Lighting
• direct lighting ray casting
• indirect lighting hierarchical radiosity

radiosity reflectance x ( direct light
indirect light )
9
Simple Input Process

• Geometric reconstruction
• several (4-5) images from different viewpoints
• geometric modelling using Rekon Poulin et al.
  98
• Reflectance reconstruction
• several (5-7) images from a single viewpoint
• different lighting conditions single light
  source at different positions
• radiance images

10
Input

• Radiance images from single viewpoint
• combining multiple images reduces artefacts of
  estimation

different lighting conditions
11
Pre-process

• Computation of approximate diffuse reflectance
  pixel by pixel
• compute individual reflectance images
• merge reflectance images using confidence values
• Initialise lighting system
• data structure
• hierarchical radiosity system

12
Reflectance Computation

• For each radiance image

reflectance radiosity / ( direct light
indirect light )
photograph
reflectance
13
Confidence Images

• Estimate confidence
• confidence quality of reflectance estimate
• create a confidence image per light source
  position
• Begin with confidence Visibility
• low in shadow regions
• Filtering process to remove unwanted effects
• low for outliers (specular effects, light tripod)

14
Merged Reflectance Computation
reflectance
confidence
merged reflectance
15
Interactive Modification Shadow Reprojection

• Direct illumination pixel by pixel
• Indirect illumination optimised radiosity
  solution

a
16
Shadow Re-projection

• photograph

simulated
17
Add/move/remove object (virtual or real)

• Visible surface changes pixel by pixel local
  update
• project bounding box of dynamic object
• localise directly affected pixels

18
Add/move/remove object (virtual or real)

• Direct lighting updates shaft structure
• localisation of visibility changes (shadows)
• accelerate visibility computation (blocker lists)

19
Add/move/remove object (virtual or real)

• Indirect illumination computed by a radiosity
  solution (optimised by the shaft structure)
• Example moving object

Position 1
Position 2
20
Real Object Removal
21
Removing Real Objects

• Use of the reflectance image (lighting effects
  removed) to generate new textures

reflectance images
22
Light Source Modification

• Insertion of a virtual light source
• computation for every pixel
• new form-factors
• new visibility
• Indirect illumination radiosity solution

23
Lighting Modification

• Insertion of virtual light

Original virtual lighting
24
Video
25
Conclusion

• Input
• data simple to acquire
• Pre-process
• data structures optimised for fast updates
• Interactive modification
• add and move virtual objects
• remove real objects
• relighting

26
Future Work

• Improve reflectance computation
• use of high dynamic range images (instead of RGB)
• better control of indirect illumination
• Allow motion of real objects
• Faster parallel computation

27
Future Work

• Remove restrictions
• diffuse reflectance Yu et al. 99
• fixed view-point