Computer Vision and Robotics Group

1
Overview of coded light projection techniques for
automatic 3D profiling

• Computer Vision and Robotics Group
• Institut dInformàtica i Aplicacions
• Jordi Pagès
• Joaquim Salvi

2
Presentation outline
Formació

• Introduction
• Coded pattern classification
• Time-multiplexing
• Spatial codification
• Direct codification
• Experimental results
• Sub-pixel matching
• Conclusions guidelines

1/36
3
Introduction shape acquisition
Shape acquisition techniques
Introduction Shape acquisition Stereovision Encodi
ng/ decoding
Non-contact
Contact
Reflective
Transmissive
Non-destructive
Destructive
Non-optical
Industrial CT
Slicing
Classification
CMM
Jointed arms
Optical
Microwave radar
Sonar
Active
Experiments
Passive
Imaging radar
Stereo
Sub-pixel matching
Shape from X
Triangulation
(Coded) Structured light
Interferometry
Stereo
Shading
Silhouettes
Texture
Motion
Conclusions
Moire
Holography
2/36
Source Brian Curless
4
Introduction passive stereovision
Introduction Shape acquisition Stereovision Encodi
ng/ decoding

• Correspondence problem
• geometric constraints ?
• search along epipolar lines
• 3D reconstruction of matched pairs by
  triangulation

Classification
Experiments
Sub-pixel matching
Conclusions
3/36
5
Introduction active stereo (coded structured
light)
Introduction Shape acquisition Stereovision Encodi
ng/ decoding

• One of the cameras is replaced by a light
  emitter
• Correspondence problem is solved by searching
  the pattern in the camera image (pattern decoding)

Classification
Experiments
Sub-pixel matching
Conclusions
4/36
6
Introduction pattern encoding/decoding (I)
Introduction Shape acquisition Stereovision Encodi
ng/ decoding

• A pattern is encoded when after projecting it
  onto a surface, a set of regions of the observed
  projection can be easily matched with the
  original pattern. Example pattern with
  two-encoded-columns
• The process of matching an image region with its
  corresponding pattern region is known as pattern
  decoding ? similar to searching correspondences
• Decoding a projected pattern allows a large set
  of correspondences to be easily found thanks to
  the a priori knowledge of the light pattern

Object scene
Pixels in red and yellow are directly matched
with the pattern columns
Codification using colors
Classification
Experiments
Sub-pixel matching
Conclusions
5/36
7
Introduction pattern encoding/decoding (II)
Introduction Shape acquisition Stereovision Encodi
ng/ decoding

• Two ways of encoding the correspondences single
  and double axis codification ? it determines how
  the triangulation is calculated
• Decoding the pattern means locating points in the
  camera image whose corresponding point in the
  projector pattern is a priori known

Double-axis encoding
Single-axis encoding
Triangulation by line-to-plane intersection
Triangulation by line-to-line intersection
Classification
Experiments
Sub-pixel matching
Conclusions
6/36
8
Coded structured light patterns a
classification proposal
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
7/36
9
Time-multiplexing
Introduction

• The time-multiplexing paradigm consists of
  projecting a series of light patterns so that
  every encoded point is identified with the
  sequence of intensities that receives
• The most common structure of the patterns is a
  sequence of stripes increasing its length by the
  time ? single-axis encoding
• Advantages
• high resolution ? a lot of 3D points
• High accuracy (order of ?m)
• Robustness against colorful objects since binary
  patterns can be used
• Drawbacks
• Static objects only
• Large number of patterns

Example 3 binary-encoded patterns which allows
the measuring surface to be divided in 8
sub-regions
Classification Time-multiplexing Spatial
codification Direct codification
Projected over time
Experiments
Pattern 3
Pattern 2
Sub-pixel matching
Pattern 1
Conclusions
8/36
10
Time-multiplexing binary codes (I)
Formació
Introduction

• Coding redundancy every edge between adjacent
  stripes can be decoded by the sequence at its
  left or at its right

Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
9/36
11
Time-multiplexing Binary codes (II)

• Every encoded point is identified by the sequence
  of intensities that receives
• n patterns must be projected in order to encode
  2n stripes

Introduction
Example 7 binary patterns proposed by Posdamer
Altschuler
Classification Time-multiplexing Spatial
codification Direct codification
Projected over time

Experiments
Pattern 3
Pattern 2
Sub-pixel matching
Pattern 1
Conclusions
Codeword of this píxel 1010010 ? identifies the
corresponding pattern stripe
10/36
12
Time-multiplexing n-ary codes (I)
Introduction

• n-ary codes reduce the number of patterns by
  increasing the number of projected intensities
  (grey levels/colours) ? increases the basis of
  the code
• The number of patterns, the number of grey levels
  or colours and the number of encoded stripes are
  strongly related ? fixing two of these parameters
  the reamaining one is obtained

Classification Time-multiplexing Spatial
codification Direct codification
Using a binary code, 6 patterns are necessary
necessary to encode 64 stripes
Experiments
Sub-pixel matching
3 patterns based on a n-ary code of 4 grey levels
(Horn Kiryati) ? 64 encoded stripes
Conclusions
11/36
13
Time-multiplexing n-ary codes (II)
Introduction

• n-ary codes reduce the number of patterns by
  increasing the number of projected intensities
  (grey levels/colours)

Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
12/36
14
Time-multiplexing Gray code Phase shifting (I)
Introduction

• A sequence of binary patterns (Gray encoded) are
  projected in order to divide the object in regions

Example three binary patterns divide the object
in 8 regions
Classification Time-multiplexing Spatial
codification Direct codification
Without the binary patterns we would not be able
to distinguish among all the projected slits

• An additional periodical pattern is projected

Experiments

• The periodical pattern is projected several
  times by shifting it in one direction in order to
  increase the resolution of the system ? similar
  to a laser scanner

Sub-pixel matching
Every slit always falls in the same region
Gührings line-shift technique
Conclusions
13/36
15
Time-multiplexing Gray code Phase shifting (II)

• A periodical pattern is shifted and projected
  several times in order to increase the resolution
  of the measurements
• The Gray encoded patterns permit to
  differentiate among all the periods of the
  shifted pattern

Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
14/36
16
Time-multiplexing hybrid methods (I)

• In order to decode an illuminated point it is
  necessary to observe not only the sequence of
  intensities received by such a point but also the
  intensities of few (normally 2) adjacent points
• The number of projected patterns reduces thanks
  to the spatial information that is taken into
  account
• The redundancy on the binary codification
• is eliminated

Introduction
Formació
Classification Time-multiplexing Spatial
codification Direct codification
1
0
Pattern 1
Hall-Holt and Rusinkiewicz technique 4 patterns
with 111 binary stripes Edges encoding 4x2 bits
(every adjacent stripe is a bit)
1
1
Pattern 2
Experiments
1
0
Pattern 3
Sub-pixel matching
1
0
Pattern 4
Conclusions
Edge codeword 10110101
15/36
17
Time-multiplexing hybrid methods (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
16/36
18
Spatial Codification
Introduction

• Spatial codification paradigm encodes a set of
  points with the information contained in a
  neighborhood (called window) around them
• The codification is condensed in a unique pattern
  instead of multiplexing it along time
• The size of the neighborhood (window size) is
  proportional to the number of encoded points and
  inversely proportional to the number of used
  colors
• The aim of these techniques is to obtain a
  one-shot measurement system ? moving objects can
  be measured

Classification Time-multiplexing Spatial
codification Direct codification

• Drawbacks
• Discontinuities on the object surface can
  produce erroneous window decoding (occlusions
  problem)
• The higher the number of used colours, the more
  difficult to correctly identify them when
  measuring non-neutral surfaces
• Maximum resolution cannot be reached

Experiments

• Advantages
• Moving objects supported
• Possibility to condense the codification to a
  unique pattern

Sub-pixel matching
Conclusions
17/36
19
Spatial codification non-formal codification (I)

• The first group of techniques that appeared used
  codification schemes with no mathematical basis
• Drawbacks
• the codification is not optimal and often
  produces ambiguities since different regions of
  the pattern are identical
• the structure of the pattern is too complex for
  a good image processing

Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Durdle et al. ? periodic pattern
Experiments
Sub-pixel matching
Maruyama and Abe ? complex structure based on
slits containing randomly placed cuts
Conclusions
18/36
20
Spatial codification non-formal codification (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
19/36
21
Spatial codification De Bruijn sequences (I)

• A De Bruijn sequence (or pseudorrandom
  sequence) of order m over an alphabet of n
  symbols is a circular string of length nm that
  contains every substring of length m exactly once
  (in this case the windows are unidimensional).
• 1000010111101001
• The De Bruijn sequences are used to define
  coloured slit patterns (single axis codification)
  or grid patterns (double axis codification)
• In order to decode a certain slit it is only
  necessary to identify one of the windows in which
  it belongs to

Introduction
m4 (window size) n2 (alphabet symbols)
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Zhang et al. 125 slits encoded with a De Bruijn
sequence of 8 colors and window size of 3 slits
Salvi et al. grid of 29?29 where a De Bruijn
sequence of 3 colors and window size of 3 slits
is used to encode the vertical and horizontal
slits
Conclusions
20/36
22
Spatial codification De Bruijn sequences (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
21/36
23
Spatial codification M-arrays (I)

• An m-array is the bidimensional extension of a
  De Bruijn sequence. Every window of w?h units
  appears only once. The window size is related
  with the size of the m-array and the number of
  symbols used
• 0 0 1 0 1 0
• 0 1 0 1 1 0
• 1 1 0 0 1 1
• 0 0 1 0 1 0

Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Example binary m-array of size 4?6 and window
size of 2?2
Experiments
Shape primitives used to represent every symbol
of the alphabet
Sub-pixel matching
M-array proposed by Vuylsteke et al. Represented
with shape primitives
Morano et al. M-arry represented with an array of
coloured dots
Conclusions
22/36
24
Spatial codification M-arrays (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
23/36
25
Direct Codification

• Every encoded pixel is identified by its own
  intensity/colour
• Since the codification is usually condensed in a
  unique pattern, the spectrum of
  intensities/colours used is very large
• Additional reference patterns must be projected
  in order to differentiate among all the projected
  intensities/colours
• Ambient lighting (black pattern)
• Full illuminated (white pattern)
• Advantages
• Reduced number of patterns
• High resolution can be teorically achieved
• Drawbacks
• Very noisy in front of reflective properties of
  the objects, non-linearities in the camera
  spectral response and projector spectrum ?
  non-standard light emitters are required in order
  to project single wave-lengths
• Low accuracy (order of 1 mm)

Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
24/36
26
Direct codification grey levels (I)

• Every encoded point of the pattern is identified
  by its intensity level

Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Every slit is identified by its own intensity
Experiments
Carrihill and Hummel Intensity Ratio Sensor fade
from balck to white
Sub-pixel matching

• Every slit must be projected using a single
  wave-length
• Cameras with large depth-per-pixel (about 11
  bits) must be used in order to differentiate all
  the projected intensities

Requirements to obtain high resolution
Conclusions
25/36
27
Direct codification grey levels (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
26/36
28
Direct codification Colour (I)
Introduction

• Every encoded point of the pattern is
  identified by its colour

Classification Time-multiplexing Spatial
codification Direct codification
Tajima and Iwakawa rainbow pattern (the rainbow
is generated with a source of white light passing
through a crystal prism)
T. Sato patterns capable of cancelling the object
colour by projecting three shifted patterns (it
can be implemented with an LCD projector if few
colours are projected ? drawback the pattern
becomes periodic in order to maintain a good
resolution)
Experiments
Sub-pixel matching
Conclusions
27/36
29
Direct codification Colour (II)
Introduction
Classification Time-multiplexing Spatial
codification Direct codification
Experiments
Sub-pixel matching
Conclusions
28/36
30
Experiments quantitative results
Technique StDev(?m) 3D Points Resolution () Patterns
Time-multiplexing Binary code Posdamer (128 stripes) 37.6 13013 11.18 9
Time-multiplexing n-ary code Horn (64 stripes) 9.6 12988 11.15 5
Time-multiplexing Phase Shift Gray code Gühring (113 slits) 4.9 27214 23.38 14
Spatical codification De Bruijn sequence De Bruijn (64 slits) 13.1 13899 11.94 1
Spatical codification De Bruijn sequence Salvi (29?29 grid) 72.3 372 0.32 1
Spatical codification M-array Morano (45?45 array) 23.6 926 0.80 1
Direct codification Colour Sato (64 stripes) 11.9 10204 8.77 3
Introduction
Classification
Experiments Quantitative results Qualitative
results
Sub-pixel matching
Conclusions
Results obtained by reconstructing 30 times two
flat panels separated by 40mm. The distance
between both panels obtained for each technique
was calculated for every reconstruction. The
Standard Deviation is indicated. () of pixels
inside a window of 515x226 of the camera image
that have been triangulated
29/36
31
Experiments qualitative results
Gühring
Time-multiplexing Posdamer (128 stripes) Horn (64 stripes) Gühring (113 slits)
Time-multiplexing
Introduction
Classification
Spatial codification De Bruijn (64 slits) Salvi (29x29 slits) Morano (45x45 dot array)
Spatial codification
Experiments Quantitative results Qualitative
results
Direct codification Sato (64 slits)
Direct codification
Sub-pixel matching
Conclusions
30/36
32
Sub-pixel matching a key point

• A key point in the accuracy of the 3D
  measurements is to locate the correspondences
  between camera image and projector pattern with
  sub-pixel coordinates
• Qualitative example

Introduction
The reflected rays of light fall between ajdacent
pixels in the camera sensor This must be taken
into account when triangulating the 3D points
Classification
Camera image
Experiments
Sub-pixel matching Stripe patterns Other patterns
Horse reconstruction with pixel-accuracy
triangulation (20000 points)
Horse reconstruction with subpixel-accuracy
triangulation (10000 points)
Conclusions
31/36
33
Sub-pixel matching stripe-patterns

• Only points belonging to the edges between
  adjacent stripes are decoded and reconstructed.
  Two possible strategies
• - Intersecting the stripe intensity profile with
  and adaptative binarization threshold (calculated
  from two additional images full illuminated and
  ambient lighting)
• - More accurate Projecting positive and negative
  patterns and intersecting the stripe profiles

Introduction
Classification
Experiments

Sub-pixel matching Stripe patterns Other patterns
Conclusions
32/36
34
Sub-pixel matching other patterns
Introduction
Classification
Experiments
Arry of dots the subpixel position of the dots
is often calculated with their mass centre ? not
very accurate since the circles are observed like
deformed ellipses due to the change of
perspective and the measuring surface
Sub-pixel matching Stripe patterns Other patterns
Slit patterns every observed slit can be
modelled with a gaussian profile and peak
detectors can be applied (like Blais Rioux
detector) ? very accurate
Conclusions
33/36
35
Conclusions
Introduction
Types of techniques ? ?
Time-multiplexing Highest resolution High accuracy Easy implementation Inapplicability to moving objects Large number of patterns
Spatial codification Can measure moving objects A unique pattern is required Lower resolution than time-multiplexing More complex decoding stage Occlusions problem
Direct codification High resolution Few patterns Very sensitive to image noise ? cameras with large depth-per-pixel required Sensitive to limited bandwith of LCD projectors ? special projector devices are usually required Inapplicability to moving objects
Classification
Experiments
Sub-pixel matching
Conclusions Guidelines
34/36
36
Guidelines
Requirements Best technique
High accuracy Highest resolution Static objects No matter the number of patterns Phase shift Gray code ? Gührings line-shift technique
High accuracy High resolution Static objects Minimum number of patterns N-ary pattern ? Horn Kiryati Caspi et al.
High accuracy Good resolution Moving objects De Bruijn pattern ? Zhang et al.
Introduction
Classification
Experiments
Sub-pixel matching
Conclusions Guidelines
35/36
37
END

• Computer Vision and Robotics Group
• Institut dInformàtica i Aplicacions
• Jordi Pagès
• Joaquim Salvi