Overview of Our Sensors For Robotics

1
Overview of Our Sensors For Robotics
2
Machine vision

• Computer vision
• To recover useful information about a scene from
  its 2-D projections.
• To take images as inputs and produce other types
  of outputs (object shape, object contour, etc.)
• Geometry Measurement Interpretation
• To create a model of the real world from images.

3
Topics

• Computer vision system
• Image enhancement
• Image analysis
• Pattern Classification

4
Related fields

• Image processing
• Transformation of images into other images
• Image compression, image enhancement
• Useful in early stages of a machine vision system
• Computer graphics
• Pattern recognition
• Artificial intelligence
• Psychophysics

5
Vision system hardware
6

• Image Processing System

7
(No Transcript)
8
Image Representation
9
Image

• Image a two-dimensional array of pixels
• The indices i, j of pixels integer values
  that specify the rows and columns in pixel values

10
(No Transcript)
11
Sampling, pixeling and quantization

• Sampling
• The real image is sampled at a finite number of
  points.
• Sampling rate image resolution
• how many pixels the digital image will have
• e.g.) 640 x 480, 320 x 240, etc.
• Pixel
• Each image sample
• At the sample point, an integer value of the
  image intensity

12

• Quantization
• Each sample is represented with the finite word
  size of the computer.
• How many intensity levels can be used to
  represent the intensity value at each sample
  point.
• e.g.) 28 256, 25 32, etc.

13
Color models

• Color models for images,
• RGB, CMY
• Color models for video,
• YIQ, YUV (YCbCr)
• Relationship between color models

14
(No Transcript)
15
6.7. Digital Cameras
16
Digital Cameras

• Technology
• CCD (charge coupled devices)
• CMOS (complementary metal oxide semiconductor)
• Resolution
• 60x80 black/white up to
• several Mega-Pixels in 32bit color
• However Embedded system has to have computing
  power to deal with this large amount of data!

17
Vision (camera framegrabber)
18
Digital Cameras

• Performance of embedded system 10 - 50 of
  standard PC

19
Interfacing Digital Cameras to CPU

• Interfacing to CPU
• Completely depends on sensor chip specs
• Many sensors provide several different
  interfacing protocols
• versatile in hardware design
• software gets very complicated
• Typically 8 bit parallel (or 4, 16, serial)
• Numerous control signals required

20
Interfacing Digital Cameras to CPU

• Digital camera sensors are very complex units.
• In many respects they are themselves similar to
  an embedded controller chip.
• Some sensors buffer camera data and allow slow
  reading via handshake (ideal for slow
  microprocessors)
• Most sensors send full image as a stream after
  start signal
• (CPU must be fast enough to read or use hardware
  buffer or DMA)
• We will not go into further details in this
  course. However, we consider camera access
  routines

21
Simplified diagram of camera to CPU interface
22
Problem with Digital Cameras

• Problem
• Every pixel from the camera causes an interrupt
• Interrupt service routines take long, since
  they need to store register contents on the stack
• Everything is slowed down
• Solution
• Use RAM buffer for image and read full image
  with single interrupt

23

• Idea
• Use FIFO as image data buffer
• FIFO is similar to dual-ported RAM, it is
  required since there is no synchronization
  between camera and CPU
• When FIFO is half full, interrupt is generated
• Interrupt service routine then reads FIFO until
  empty
• (Assume delay is small enough to avoid FIFO
  overrun)

24
(No Transcript)
25
(No Transcript)
26
Bayer Pattern
27
De-Mosaic
28
(No Transcript)
29
Conversion in Digital Cameras

• Bayer Pattern
• Output format of most digital cameras
• Note
• 2x2 pattern is not spatially located in a
  single point!
• Can be simply converted to RGB (drop one green
  byte)
• 160x120 Bayer ? 80x60 RGB
• Can be better converted using demosaicing
  technique
• 160x120 Bayer ? 160x120 RGB

30
(No Transcript)
31

• CMUCAM2 CAMERA www.seattlerobotics.com
• The camera can track user defined color blobs at
  up to 50 fps (frames per second)
• Track motion using frame differencing at 26 fps
• Find the centroid of any tracking data
• Gather mean color and variance data
• Gather a 28 bin histogram of each color channel
• Manipulate horizontal pixel differenced images
• Arbitrary image windowing
• Adjust the cameras image properties

This camera can do a lot of processing
32
This camera can do a lot of processing

• Dump a raw image
• Up to 160 X 255 resolution
• Support multiple baud rates
• Control 5 servos outputs
• Slave parallel image processing mode off of
  single camera bus
• Automatically use servos to do two axis color
  tracking
• B/W analog video output (Pal or NTSC)
• Flexible output packet customization
• Multiple pass image processing on a buffered image

33
(No Transcript)
34
Vision Guided Robotics

• and Applications in Industry and Medicine

35
Contents

• Robotics in General
• Industrial Robotics
• Medical Robotics
• What can Computer Vision do for Robotics?
• Vision Sensors
• Issues / Problems
• Visual Servoing
• Application Examples
• Summary

36
Industrial Robot vs Human

• Robot Advantages
• Strength
• Accuracy
• Speed
• Does not tire
• Does repetitive tasks
• Can Measure

• Human advantages
• Intelligence
• Flexibility
• Adaptability
• Skill
• Can Learn
• Can Estimate

Robot needs vision
37
Industrial Robot

• Requirements
• Accuracy
• Tool Quality
• Robustness
• Strength
• Speed
• Price Production Cost
• Maintenance

Production Quality
38
Medical (Surgical) Robot

• Requirements
• Safety
• Accuracy
• Reliability
• Tool Quality
• Price
• Maintenance
• Man-Machine Interface

39
What can Computer Vision do for (industrial and
medical) Robotics?

• Accurate Robot-Object Positioning
• Keeping Relative Position under Movement
• Visualization / Teaching / Telerobotics
• Performing measurements
• Object Recognition
• Registration

Visual Servoing
40
Vision Sensors

• Single Perspective Camera
• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)
• Laser Scanner
• Omnidirectional Camera
• Structured Light Sensor

41
Vision Sensors

• Single Perspective Camera

Single projection
42
Vision Sensors

• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)

43
Vision Sensors

• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)

44
Vision Sensors

• Multiple Perspective Cameras (e.g. Stereo Camera
  Pair)

45
Vision Sensors

• Laser Scanner

46
Vision Sensors

• Laser Scanner

47
Vision Sensors

• Omnidirectional Camera

48
Vision Sensors

• Omnidirectional Camera

49
Vision Sensors

• Structured Light Sensor

Figures from PRIP, TU Vienna
50
Issues/Problems of Vision Guided Robotics

• Measurement Frequency
• Measurement Uncertainty
• Occlusion, Camera Positioning
• Sensor dimensions

51
Visual Servoing

• Vision System operates in a closed control loop.
• Better Accuracy than Look and Move systems

Figures from S.Hutchinson A Tutorial on Visual
Servo Control
52
Visual Servoing

• Example Maintaining relative Object Position

Figures from P. Wunsch and G. Hirzinger.
Real-Time Visual Tracking of 3-D Objects with
Dynamic Handling of Occlusion
53
Camera Configurations for Visual Servoing
End-Effector Mounted
Fixed
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
54
Visual Servoing Architectures
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
55
Position-based vs Image Based control in Visual
Servoing

• Position based
• Alignment in target coordinate system
• The 3D structure of the target is rconstructed
• The end-effector is tracked
• Sensitive to calibration errors
• Sensitive to reconstruction errors
• Image based
• Alignment in image coordinates
• No explicit reconstruction necessary
• Insensitive to calibration errors
• Only special problems solvable
• Depends on initial pose
• Depends on selected features

End-effector
target
Image of end effector
Image of target
56
EOL and ECL control in Visual Servoing

• EOL endpoint open-loop only the target is
  observed by the camera
• ECL endpoint closed-loop target as well as
  end-effector are observed by the camera

EOL
ECL
57
Visual Servoing

• Position Based Algorithm
• Estimation of relative pose
• Computation of error between current pose and
  target pose
• Movement of robot
• Example point alignment

p1
p2
58
Visual Servoing

• Position based point alignment
• Goal bring e to 0 by moving p1
• e p2m p1m
• u k(p2m p1m)
• pxm is subject to the following measurement
  errors sensor position, sensor calibration,
  sensor measurement error
• pxm is independent of the following errors end
  effector position, target position

59
Visual Servoing

• Image based point alignment
• Goal bring e to 0 by moving p1
• e u1m v1m u2m v2m
• uxm, vxm is subject only to sensor measurement
  error
• uxm, vxm is independent of the following
  measurement errors sensor position, end effector
  position, sensor calibration, target position

p1
p2
u1
v1
v2
u2
d1
d2
c1
c2
60
Visual Servoing

• Example Laparoscopy

Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
61
Visual Servoing

• Example Laparoscopy

Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
62
Registration

• Registration of CAD models to scene features

Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
63
Registration

• Registration of CAD models to scene features

Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
64
Summary on tracking and servoing

• Computer Vision provides accurate and versatile
  measurements for robotic manipulators
• With current general purpose hardware, depth and
  pose measurements can be performed in real time
• In industrial robotics, vision systems are
  deployed in a fully automated way.
• In medicine, computer vision can make more
  intelligent surgical assistants possible.

65
Omnidirectional Vision Systems

• CABOTO Robots task
• Building a topological map of an unknown
  environment
• Sensor
• Omnidirectional vision system
• Works aim
• Prove effectiveness of omnidirectional sensors
  for Spatial Semantic Hierarchy SSH

66
Spatial Semantic Hierarchy...

• ... A model of the human knowledge of large
  spaces

• Layers
• Sensory Level
• Control Level
• Causal Level
• Topological Level
• Metrical Level

Interface with the robots sensory system
Control Laws, Transition of State,
Distinctiveness Measure
Minimal set of Places, Paths and Regions
View, Action, Distinct Place Abstracts Discrete
from Continous
Distance, Direction, Shape Useful, but seldom
essential
67
Tracking

• Instrument tracking in laparoscopy

Figures from Wei A Real-time Visual Servoing
System for Laparoscopic Surgery
68
Omnidirectional Camera

• Composed of
• Standard Color Camera
• Convex Mirror
• Perspex Cylinder

69
Pros e Cons

• Advantages
• Wide vision field
• High speed
• Vertical Lines
• Rotational Invariance

• Disadvantages
• Low Resolution
• Distortions
• Low readability

70
Omnidirectional Vision and SSH

• View Omnidirectional image

• Exploring around the block

• Robot should discriminate between turns and
  travels
• We need an Effective Distinctiveness measure

71
Assumptions for vision system

• Man-made environment
• Floor flat and horizontal
• Wall and objects surfaces are vertical
• Static objects
• Constant Lighting
• Robot translates or rotates
• No encoders

72
Features and Events

• Feature
• Vertical Edges
• Events
• A new edge
• An edge disappears
• Two edges 180 apart
• Two pairs of edges 180 apart

73
Experiments

• Tasks of Caboto robot
• Navigation
• Map building
• Techniques
• Edge detection
• Colour marking

74
Cabotos Images
75
(No Transcript)
76
Results

• Correct tracking of edges
• Recognition of actions
• Calculation of the turn angle

The path segmentation
77
Mirror Design
Mirror shape should depend on robot task!

• Design custom mirror profile
• Maximise resolution in ROIs

Mirror Profile
78
The new mirror
79
Conclusion on Omnivision camera

• Omnidirectional vision sensor is a good sensor
  for map building with SSH
• Motion of the robot was estimated without active
  vision
• The use of a mirror designed for this application
  will improve the system

80
Omnidirectional Cameras

• Compound-eye camera
• (from Univ. of Maryland, College Park. )
• Panoramic cameras (from Apple)
• Omnidirectional cameras
• (from University of Picardie - France)

81
Student info.

• of lab marks can be deducted if rules and
  regulation are not followed
• ex by not cleaning up your bench or sliding
  your chairs back
• underneath bench top.
• For more technical information on boards, devices
  and sensors check out my web page at
  www.site.uottawa.ca/alan
• Students are responsible for their own extra
  parts ex if you want to add a sensor or device
  that the dept. doesnt have you are responsible
  for the purchase and delivery of that part, on
  rare occasion did the school purchase those
  parts.
• Back packs off bench tops
• TAs will have student based on station
• Important issue regarding the design of a new
  project is to do a current analysis before the
  start of your design
• Setup a leader among your team so that you are
  better organized
• Do not wait, before starting your project start
  now !
• Prepare yourself before coming to the lab
• It doesnt work ! Ask yourself is it software or
  hardware, use the scope to trouble shoot
• Fuses keeps on blowing, stop and do some
  investigation.
• Do not cut any servo, battery and other device
  wire connectors. If you must please come and see
  me
• No design must exceed 50 volts, ex do not work
  with 120 volts AC
• I can give you what I have regarding metal, wood
  and plastic recycled pieces and do some cuts or
  holes with my band saw and drill press for you,
• PLEASE DO NOT ask me to barrow my tools. If you
  need to do a task with a special tool that I have
  then I shall do it for you.

82
Problems for students

1. Hardware and software components of a vision
   system for a mobile robot
2. Image representation for intelligent processing
3. Sampling, pixeling and Quantization
4. Color models
5. Types of digital cameras.
6. Interfacing digital cameras to CPU.
7. Problems with cameras.
8. Bayer Patterns and conversion.
9. What is good about CMUCAM?
10. Use of vision in industrial robots.
11. Use of multiple-perspective cameras.
12. Use of omnivision cameras.
13. Types of visual servoing.
14. Applications of visual servoing
15. Visual servoing in surgery
16. Explain tracking applications of vision.

83
References

• Photos ,Text and Schematics Information
• www.acroname.com
• www.lynxmotion.com
• www.drrobot.com
• Alan Stewart

• Dr. Gaurav Sukhatme
• Thomas Braunl
• Students 2002, class 479

• E. Menegatti, M. Wright, E. Pagello