Title: Overview of Our Sensors For Robotics
1Overview of Our Sensors For Robotics
2Machine vision
- Computer vision
- To recover useful information about a scene from
its 2-D projections. - To take images as inputs and produce other types
of outputs (object shape, object contour, etc.) - Geometry Measurement Interpretation
- To create a model of the real world from images.
3Topics
- Computer vision system
- Image enhancement
- Image analysis
- Pattern Classification
4 Related fields
- Image processing
- Transformation of images into other images
- Image compression, image enhancement
- Useful in early stages of a machine vision system
- Computer graphics
- Pattern recognition
- Artificial intelligence
- Psychophysics
5 Vision system hardware
6 7(No Transcript)
8Image Representation
9 Image
- Image a two-dimensional array of pixels
- The indices i, j of pixels integer values
that specify the rows and columns in pixel values
10(No Transcript)
11 Sampling, pixeling and quantization
- Sampling
- The real image is sampled at a finite number of
points. - Sampling rate image resolution
- how many pixels the digital image will have
- e.g.) 640 x 480, 320 x 240, etc.
- Pixel
- Each image sample
- At the sample point, an integer value of the
image intensity
12- Quantization
- Each sample is represented with the finite word
size of the computer. - How many intensity levels can be used to
represent the intensity value at each sample
point. - e.g.) 28 256, 25 32, etc.
13 Color models
- Color models for images,
- RGB, CMY
- Color models for video,
- YIQ, YUV (YCbCr)
- Relationship between color models
14(No Transcript)
156.7. Digital Cameras
16Digital Cameras
- Technology
- CCD (charge coupled devices)
- CMOS (complementary metal oxide semiconductor)
- Resolution
- 60x80 black/white up to
- several Mega-Pixels in 32bit color
- However Embedded system has to have computing
power to deal with this large amount of data!
17Vision (camera framegrabber)
18Digital Cameras
- Performance of embedded system 10 - 50 of
standard PC
19Interfacing Digital Cameras to CPU
- Interfacing to CPU
- Completely depends on sensor chip specs
- Many sensors provide several different
interfacing protocols - versatile in hardware design
- software gets very complicated
- Typically 8 bit parallel (or 4, 16, serial)
- Numerous control signals required
20Interfacing Digital Cameras to CPU
- Digital camera sensors are very complex units.
- In many respects they are themselves similar to
an embedded controller chip. - Some sensors buffer camera data and allow slow
reading via handshake (ideal for slow
microprocessors) - Most sensors send full image as a stream after
start signal - (CPU must be fast enough to read or use hardware
buffer or DMA) - We will not go into further details in this
course. However, we consider camera access
routines
21Simplified diagram of camera to CPU interface
22Problem with Digital Cameras
- Problem
- Every pixel from the camera causes an interrupt
- Interrupt service routines take long, since
they need to store register contents on the stack - Everything is slowed down
- Solution
- Use RAM buffer for image and read full image
with single interrupt
23- Idea
- Use FIFO as image data buffer
- FIFO is similar to dual-ported RAM, it is
required since there is no synchronization
between camera and CPU - When FIFO is half full, interrupt is generated
- Interrupt service routine then reads FIFO until
empty - (Assume delay is small enough to avoid FIFO
overrun)
24(No Transcript)
25(No Transcript)
26Bayer Pattern
27De-Mosaic
28(No Transcript)
29Conversion in Digital Cameras
- Bayer Pattern
- Output format of most digital cameras
- Note
- 2x2 pattern is not spatially located in a
single point! - Can be simply converted to RGB (drop one green
byte) - 160x120 Bayer ? 80x60 RGB
- Can be better converted using demosaicing
technique - 160x120 Bayer ? 160x120 RGB
30(No Transcript)
31- CMUCAM2 CAMERA www.seattlerobotics.com
- The camera can track user defined color blobs at
up to 50 fps (frames per second) - Track motion using frame differencing at 26 fps
- Find the centroid of any tracking data
- Gather mean color and variance data
- Gather a 28 bin histogram of each color channel
- Manipulate horizontal pixel differenced images
- Arbitrary image windowing
- Adjust the cameras image properties
This camera can do a lot of processing
32This camera can do a lot of processing
- Dump a raw image
- Up to 160 X 255 resolution
- Support multiple baud rates
- Control 5 servos outputs
- Slave parallel image processing mode off of
single camera bus - Automatically use servos to do two axis color
tracking - B/W analog video output (Pal or NTSC)
- Flexible output packet customization
- Multiple pass image processing on a buffered image
33(No Transcript)
34Vision Guided Robotics
- and Applications in Industry and Medicine
35Contents
- Robotics in General
- Industrial Robotics
- Medical Robotics
- What can Computer Vision do for Robotics?
- Vision Sensors
- Issues / Problems
- Visual Servoing
- Application Examples
- Summary
36Industrial Robot vs Human
- Robot Advantages
- Strength
- Accuracy
- Speed
- Does not tire
- Does repetitive tasks
- Can Measure
- Human advantages
- Intelligence
- Flexibility
- Adaptability
- Skill
- Can Learn
- Can Estimate
Robot needs vision
37Industrial Robot
- Requirements
- Accuracy
- Tool Quality
- Robustness
- Strength
- Speed
- Price Production Cost
- Maintenance
Production Quality
38Medical (Surgical) Robot
- Requirements
- Safety
- Accuracy
- Reliability
- Tool Quality
- Price
- Maintenance
- Man-Machine Interface
39What can Computer Vision do for (industrial and
medical) Robotics?
- Accurate Robot-Object Positioning
- Keeping Relative Position under Movement
- Visualization / Teaching / Telerobotics
- Performing measurements
- Object Recognition
- Registration
Visual Servoing
40Vision Sensors
- Single Perspective Camera
- Multiple Perspective Cameras (e.g. Stereo Camera
Pair) - Laser Scanner
- Omnidirectional Camera
- Structured Light Sensor
41Vision Sensors
- Single Perspective Camera
Single projection
42Vision Sensors
- Multiple Perspective Cameras (e.g. Stereo Camera
Pair)
43Vision Sensors
- Multiple Perspective Cameras (e.g. Stereo Camera
Pair)
44Vision Sensors
- Multiple Perspective Cameras (e.g. Stereo Camera
Pair)
45Vision Sensors
46Vision Sensors
47Vision Sensors
48Vision Sensors
49Vision Sensors
Figures from PRIP, TU Vienna
50Issues/Problems of Vision Guided Robotics
- Measurement Frequency
- Measurement Uncertainty
- Occlusion, Camera Positioning
- Sensor dimensions
51Visual Servoing
- Vision System operates in a closed control loop.
- Better Accuracy than Look and Move systems
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
52Visual Servoing
- Example Maintaining relative Object Position
Figures from P. Wunsch and G. Hirzinger.
Real-Time Visual Tracking of 3-D Objects with
Dynamic Handling of Occlusion
53Camera Configurations for Visual Servoing
End-Effector Mounted
Fixed
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
54Visual Servoing Architectures
Figures from S.Hutchinson A Tutorial on Visual
Servo Control
55Position-based vs Image Based control in Visual
Servoing
- Position based
- Alignment in target coordinate system
- The 3D structure of the target is rconstructed
- The end-effector is tracked
- Sensitive to calibration errors
- Sensitive to reconstruction errors
- Image based
- Alignment in image coordinates
- No explicit reconstruction necessary
- Insensitive to calibration errors
- Only special problems solvable
- Depends on initial pose
- Depends on selected features
End-effector
target
Image of end effector
Image of target
56EOL and ECL control in Visual Servoing
- EOL endpoint open-loop only the target is
observed by the camera - ECL endpoint closed-loop target as well as
end-effector are observed by the camera
EOL
ECL
57Visual Servoing
- Position Based Algorithm
- Estimation of relative pose
- Computation of error between current pose and
target pose - Movement of robot
- Example point alignment
p1
p2
58Visual Servoing
- Position based point alignment
- Goal bring e to 0 by moving p1
- e p2m p1m
- u k(p2m p1m)
- pxm is subject to the following measurement
errors sensor position, sensor calibration,
sensor measurement error - pxm is independent of the following errors end
effector position, target position
59Visual Servoing
- Image based point alignment
- Goal bring e to 0 by moving p1
- e u1m v1m u2m v2m
- uxm, vxm is subject only to sensor measurement
error - uxm, vxm is independent of the following
measurement errors sensor position, end effector
position, sensor calibration, target position
p1
p2
u1
v1
v2
u2
d1
d2
c1
c2
60Visual Servoing
Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
61Visual Servoing
Figures from A.Krupa Autonomous 3-D Positioning
of Surgical Instruments in Robotized Laparoscopic
Surgery Using Visual Servoing
62Registration
- Registration of CAD models to scene features
Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
63Registration
- Registration of CAD models to scene features
Figures from P.Wunsch Registration of CAD-Models
to Images by Iterative Inverse Perspective
Matching
64Summary on tracking and servoing
- Computer Vision provides accurate and versatile
measurements for robotic manipulators - With current general purpose hardware, depth and
pose measurements can be performed in real time - In industrial robotics, vision systems are
deployed in a fully automated way. - In medicine, computer vision can make more
intelligent surgical assistants possible.
65Omnidirectional Vision Systems
- CABOTO Robots task
- Building a topological map of an unknown
environment - Sensor
- Omnidirectional vision system
- Works aim
- Prove effectiveness of omnidirectional sensors
for Spatial Semantic Hierarchy SSH
66Spatial Semantic Hierarchy...
- ... A model of the human knowledge of large
spaces
- Layers
- Sensory Level
- Control Level
- Causal Level
- Topological Level
- Metrical Level
Interface with the robots sensory system
Control Laws, Transition of State,
Distinctiveness Measure
Minimal set of Places, Paths and Regions
View, Action, Distinct Place Abstracts Discrete
from Continous
Distance, Direction, Shape Useful, but seldom
essential
67Tracking
- Instrument tracking in laparoscopy
Figures from Wei A Real-time Visual Servoing
System for Laparoscopic Surgery
68Omnidirectional Camera
- Composed of
- Standard Color Camera
- Convex Mirror
- Perspex Cylinder
69Pros e Cons
- Advantages
- Wide vision field
- High speed
- Vertical Lines
- Rotational Invariance
- Disadvantages
- Low Resolution
- Distortions
- Low readability
70Omnidirectional Vision and SSH
- View Omnidirectional image
- Exploring around the block
- Robot should discriminate between turns and
travels - We need an Effective Distinctiveness measure
71Assumptions for vision system
- Man-made environment
- Floor flat and horizontal
- Wall and objects surfaces are vertical
- Static objects
- Constant Lighting
- Robot translates or rotates
- No encoders
72Features and Events
- Feature
- Vertical Edges
- Events
- A new edge
- An edge disappears
- Two edges 180 apart
- Two pairs of edges 180 apart
73Experiments
- Tasks of Caboto robot
- Navigation
- Map building
- Techniques
- Edge detection
- Colour marking
74Cabotos Images
75(No Transcript)
76Results
- Correct tracking of edges
- Recognition of actions
- Calculation of the turn angle
The path segmentation
77Mirror Design
Mirror shape should depend on robot task!
- Design custom mirror profile
- Maximise resolution in ROIs
Mirror Profile
78The new mirror
79Conclusion on Omnivision camera
- Omnidirectional vision sensor is a good sensor
for map building with SSH - Motion of the robot was estimated without active
vision - The use of a mirror designed for this application
will improve the system
80Omnidirectional Cameras
- Compound-eye camera
- (from Univ. of Maryland, College Park. )
- Panoramic cameras (from Apple)
- Omnidirectional cameras
- (from University of Picardie - France)
81Student info.
- of lab marks can be deducted if rules and
regulation are not followed - ex by not cleaning up your bench or sliding
your chairs back - underneath bench top.
- For more technical information on boards, devices
and sensors check out my web page at
www.site.uottawa.ca/alan - Students are responsible for their own extra
parts ex if you want to add a sensor or device
that the dept. doesnt have you are responsible
for the purchase and delivery of that part, on
rare occasion did the school purchase those
parts. - Back packs off bench tops
- TAs will have student based on station
- Important issue regarding the design of a new
project is to do a current analysis before the
start of your design - Setup a leader among your team so that you are
better organized - Do not wait, before starting your project start
now ! - Prepare yourself before coming to the lab
- It doesnt work ! Ask yourself is it software or
hardware, use the scope to trouble shoot - Fuses keeps on blowing, stop and do some
investigation. - Do not cut any servo, battery and other device
wire connectors. If you must please come and see
me - No design must exceed 50 volts, ex do not work
with 120 volts AC - I can give you what I have regarding metal, wood
and plastic recycled pieces and do some cuts or
holes with my band saw and drill press for you, - PLEASE DO NOT ask me to barrow my tools. If you
need to do a task with a special tool that I have
then I shall do it for you.
82Problems for students
- Hardware and software components of a vision
system for a mobile robot - Image representation for intelligent processing
- Sampling, pixeling and Quantization
- Color models
- Types of digital cameras.
- Interfacing digital cameras to CPU.
- Problems with cameras.
- Bayer Patterns and conversion.
- What is good about CMUCAM?
- Use of vision in industrial robots.
- Use of multiple-perspective cameras.
- Use of omnivision cameras.
- Types of visual servoing.
- Applications of visual servoing
- Visual servoing in surgery
- Explain tracking applications of vision.
83References
- Photos ,Text and Schematics Information
- www.acroname.com
- www.lynxmotion.com
- www.drrobot.com
- Alan Stewart
- Dr. Gaurav Sukhatme
- Thomas Braunl
- Students 2002, class 479
-
- E. Menegatti, M. Wright, E. Pagello