From 2D to 3D and Stereo Machine Vision

1
From 2D to 3D andStereo Machine Vision

• Helge Jordfald
• Tordivel AS

2
3D Machine Vision Status

• 3D Machine Vision is currently only for special
  tasks and expensive
• 3D machine vision can be marketed as 2.5 D when a
  3D image is not created
• It is difficult to select the appropriate 3D
  method for a specific application due to
  diversity in solution space

3
Scorpion Vision 3D Strategy

• Work hard to gain experience and exploit the new
  3D solution space
• Teach our customers 3D
• Create 3D training material and examples for self
  study
• Establish a Scorpion 3D vocabulary

4
Scorpion Vision 3D Vision

• Take advantage of the smart Scorpion Framework to
  implement the best for 3D Machine Vision software
  solution
• Scorpion Vision 3D shall consist of
• 3D camera calibration, 3D Images, 3D
  Visualisation, numerous ways to create and
  analyse 3D images
• 3D image creation can be stripelight, laser line,
  stereo vision, laser grid and more
• 3D shall be an natural enhancement of Scorpion
  Vision 2D

5
Content

• Some history and the basic theory
• 2D Vision system
• 2D techniques for handling heights variation
• 3D Vision System
• Basic 3D Vision techniques
• Advanced 3D vision techniques to be continued
• Point Cloud Stereo Vision
• Holography structured light

6
Camera obscura Latin for dark room

• Light is admitted through a narrow hole into a
  dark chamber.
• An inverted image is formed on the opposing wall.
• A device known for at least 2000 years
• Chinese philosopher Mo-Ti (5th century BC)
• Aristotle (384-322 BC)
• Leonardo Da Vinci (1490)
• German astronomer Johannes Kepler (early 17th
  century)

7
Descriptive Geometry

• Gaspard Monge(1746-1818), the father of
  descriptive geometry, developed a graphical
  protocol that creates three-dimensional virtual
  space on a two-dimensional plane.
• Monge became a scientific and mathematical aide
  to Napoleon during his reign as General and
  Emperor of France.

8
Descriptive Geometry

• Defined as the projection of three-dimensional
  figures onto a two-dimensional plane
• The purpose is to allow geometric manipulations
  to determine
• lengths, angles, shapes
• and other descriptive information concerning the
  figures

9
Projection

• When representing a 3-D object on the 2-D sheet
  of paper, the number of dimensions is reduced
  from 3 to 2.
• The general process of reducing the number of
  dimensions of a given object is called
  projection.
• Two different ways of doing this according to the
  position of observer
• Parallel Projections infinitely far away from
  object
• Perspective/Central Projections close to the
  object

10
Parallel projection

• Descriptive Geometry is based on Parallel
  Projection, in most cases parallel, orthogonal
  projection.

11
Orthographic projection

• Orthographic projection is a means of
  representing a three-dimensional (3D) object in
  two dimensions (2D). It uses multiple views of
  the object, from points of view rotated about the
  object's center through increments of 90.

12
Perspective (Central) projection

• In this case, where the observer is relatively
  close to the object, the projectors form a cone
  of projectors.

13
Central Projection Pin Hole Camera
14
How we see the world Image Plane
15
The model we use
16
2D Vision System

• Optical System calibration
• Camera (CCD and electronics)
• Lens
• Camera/Lens position
• 2D Camera Model
• Calibrate the camera in one plane

17
2D Camera Calibration step 1

• Original image

• After eliminating lens distortion

18
Camera calibration step 2

• Camera positioned in perspective

• Camera corrected for perspective

Calibration plane
19
Intermediate result
True image plane

• Creates a Virtual pinhole camera with a
  mathematical model between calibration plane and
  true image plane
• The 2D model represents only the object in one
  flat plane

Calibration plane
20
Camera calibration optional step 3

• Image plane moved/scaled to object plane

21
2D Camera model

• Object points in calibration plane is only
  correct in the image plane

True Image plane
Image plane
Calibration plane
22
Challenges with 2D Camera Model

• Product with different heights or in different
  layers

23
Multiple 2D camera models / references
External Reference tool
3 different 2D camera calibrations, one model
mustbe selected based on the height of the object
Image plane 3
Reference plane 3
Image plane 2
Reference plane 2
Image plane 1
Reference plane 1
Possible to use linear approximationin between
the 2D camera models
24
Challenge 2 Inclined Geometry

• Cannot be described with a 2D Model

25
Challenge 3 Disorganised Products

• Products with varying angle relative to the
  calibration plane
• Cannot be described by a 2D Model

26
Laser Triangulation

• A laser system projects a spot or line of light
  to the target, and a camera system takes an image
  of its reflection.
• The position of the spot / line described the
  elevation of the object reflecting the spot/
  line.

Triangular geometry betweenlaser, camera and
reflection point
27
Laser triangulation calibration
Measurement setup
Measure point coordinates in imageand enter them
real coordinatesin External Reference Tool

Virtual camera in Laser plane
Laser line in image
Object in laser plane
Calibration object
28
3D ScanningCombine multiple laserline to create
a 3D Image

• Require scanning to move one Virtual camera (to
  get parallel laser lines)
• X and Y are still only with a 2D camera model
• The 3D image is represented as a 2D height map

29
Summary 2D Camera Calibration

• Describe Lens Distortion
• Create a virtual camera with image plane equal to
  calibration plane
• Use a calibration grid and the Calibrator Tool
• Use multiple reference system to describe height
  variations
• Create multiple 2d references
• Use External Reference tool
• Combine 2d references
• Use Python to select the best 2D plane,
• Use Python and Change Reference tool to
  interpolate between the different 2D planes
• A simple concept to handle inclined objects
• If possible locate 4 points in a plane
• Use the four points to describe the inclined
  plane with External Reference tool
• Note Inclined object shall be handled in 3D

30
3D vision system

• 3D camera model
• 3D camera calibration
• 3D reference systems
• Create a new 3D reference system
• Moving from one 3D reference system to another -
  Robotics
• 3D (Monocular) reconstruction
• Stereo vision

31
3D camera model

• True central/perspective projection to 2D image
  plane
• Each point in the 3D will be projected correctly
  to the Image Plane

Image plane
32
3D camera calibration

• Measure minimum 7 points points in the 2D image
  with different x, y and z

33
3D Objects

• 3D point
• x, y, z coordinates
• 3D line
• Two 3d points
• 3D angle
• Angle between two 3D lines
• 3D plane (frame)
• Origin x, y, z
• Rotation around axis
• Rx, Ry, Rz

34
Euler

• Leonhard Euler (1707 1783) Swiss mathematician
  and physicist. He published more papers than any
  other mathematician in history.
• Introduced much of the modern mathematical
  terminology and notation and also renowned for
  his work in mechanics, optics, and astronomy.

35
Euler Angles
36
Fixed Angles (PRY)
37
Fixed angles versus Euler angles

• Three rotations taken about fixed axes (Fixed
  Angles) yield the same final orientation as the
  same three rotations taken in an opposite order
  about the axes of the moving frame (Euler Angles)

38
Move the 3D Reference System

• The Image Plane can be moved to any position in
  the 3D model using Change Reference 3D tool
• Specify new origin and rotationaroundx, y, z

39
Make the 3D Reference relative to an object

• Measure 4 or more points with known height
• Use the Tool ExternalPoint3D to add the height
• Use the Tool ReferencefromPoints3D to create the
  3D reference system from the 3D points

40
Definition of a 3D plane

• A 3D point defines the Origin
• The normal vector of the new 3D plane (Z axis) is
  moved to the origin of the original reference
  system
• The projection of the normal vector to x, y, z
  axis defines the rotation of the new 3D plane (A
  normal vector has a nominal magnitude of 1)

41
Stereo vision

• What is stereo vision?
• How to create a stereo vision system?
• Stereo vision processing techniques
• Key issues related to accuracy

42
Stereo Vision Concept

• The slightly different perspectives from which 2
  or more cameras perceive the world lead to
  different images with relative displacements of
  objects disparities - in the different
  monocular views of the scene
• The size and direction of the disparities of an
  object is a measure of its relative depth
  absolute depth-information can be obtained if the
  geometry of the imaging system is known

43
Scorpion 3D Camera Calibration

• Remove lens distortion with Calibrator
• Describe 3D Camera Models using
  ExternalReference3D

44
Stereo Vision Multiple images working in the
same 3D space

• Multiple cameras or images calibrated in the same
  3D space
• Creation of a common 3D Space
• 1. Calibrated multiple cameras with the same 3D
  object
• 2. Know the displacment of the camera or object
  in time dynamic generation of common 3D space

45
Locate a 3D pointHow does it work in Scorpion?

• Using a Blob to find the centre of gravity of an
  object in all cameras
• Use the tool Locate3D to get the 3D position of
  centre of gravity

46
Finding a 3D Plane with Stereo Vision

• Find the corners in each image
• Same tool set and use tool template class
• Use the tool Locate3D to link the points from
  each camera to get eth new 3D plane

47
Elements deciding what technique is required

• A priori knowledge (from what is before)
• Position of object, geometry and dimensions
• What information we need to measure for solving
  the task

48
3D camera calibration removes the need for
multiple 2D references

• For each height we can create a new image plane
  by using Change Reference 3D tool. Only the
  height input (translation z) is necessary

X
49
3D camera model describes how an pixel moves in
space!

• With a 3D model you can handled inclined object
  and unordered objects
• One solution
• Measure the position of each corner
• Add Z value (a priori knowledge)
• Create the 3D plane

50
End

• 3D is just smarter than 2D
• With Scorpion Vision the step is a natural
  evolution