Visionbased Occupancy Modeling

1
Vision-based Occupancy Modeling
Dmitry O. Gorodnichy William W.
ArmstrongDepartment of Computing
ScienceUniversity of Alberta http//www.cs.ualber
ta.ca/dmitri
2
Outline

• World Modeling
• Why Vision ? Why Occupancy Approach ?
• Problems and alternative solutions
• Designing fast and affordable 3D sensor
• Visual sensor implementation Calculating
  reliability values
• Combining dependent range data
• Representing the occupancy
• Memory consumption Utility to application
• Putting into a framework. Conclusions.

3
World Exploration Task
An agent equipped with range sensors is put in
unknown environment. The agent tries to
understand what is around it, by building a
model of the environment, in order to accomplish
a task.

• Quality of the model is governed by
• the task the agent has to fulfill
• the quality of the sensor

4
Types of World Modeling

• Slow, but precise (NRC,ULaval)
• For virtual environments
• cyber-cities, 3D mapping
• Crude, but fast
• In mobile robotics
• Exploring, finding objects

A test-bed problem Find a hidden target (green
triangle) in an unknown room by navigating and
exploring
5
Range sensors (Why Vision?)

• Sonars (CMU,UMich,NCAR, ) slow, not for 3D
  acquisition
• Laser-based (NRC,Laval, , UofA) expensive
  and/or slow
• Other radars, lidars and infrared camera
• Vision-based sensors (UBC,McGill,UBonn,CMU,UPenn,
  )
• 1) Good for building 3D models and large scale
  models.
• 2) Good for surveillance and recognition
• 3) Fast and affordable
• Depends on the quality of the stereo system
• (The more robust the stereo, the slower and/or
  more expensive it is)

6
Occupancy Approach

• When sensor data are not reliable
• Calculate evidence m that a point r in space is
  occupied by building
• the Occupancy Function from excessive amount of
  range data
• Range data are pairs depth confidence values
• To be used when
• Unreliable sensors are used
• No geometry constraints can be implied
• Environment is changing
• Time is critical (e.g. in mobile robotics)

7
Occupancy Approach (cntd)

• Previous work (1982-1999)
• Moravec, Elfes, Thrun, Borenstein, Moreno,
  Payeur, Elsner

• Questions
• How to assign evidence to a point ?
• To the observed point ?
• To the points in the vicinity of the observed
  point ?
• How to build ?
• How to combine evidences ?
• How to represent the occupancy function ?
• How to use it ?

8
From Range Data to Models
Work example vision-based navigation
9
Visual Sensor Design

• Previous work
• For occupancy modeling UBC, CMU, UBonn,
• For geometry-based modeling McGill, Bochum,
  INRIA,
• What do we want ?
• Low-cost and fast 3D range sensor
• To acquire data all around (but not too much of
  data)
• Easy assignment of confidence values
• What does it mean?
• Using one off-the-shelf camera only
• Avoiding long computations

10
Stereo Rig Design
Single-camera stereo
Where problems come from?
11
Limitations of Camera

• What would you tell about the quality of this
  image?

(Those are green rectangles on white wall as
observes by a camera)
12
Complexity of Environment

• What would you tell about the distinctiveness of
  features?

(Those are features selected for tracking in a
160 x 120 image)
13
Depth Calculation Details

• RGB, 160 x 120 images, with no rectification

• Perspective camera model focal
  length F120
• Features
• are tracked along the epipolar line (/ a pixel)
  
• are those which are easy to track Iy gt Ithresh
  
• on margin are discarded

14
Sensor model

• Sensor model defines the evidence/probability
  that a point in space is occupied given a
  range sensor measurement
• Questions
• How to calculate this value for the observed
  point ?
• How to calculate this value for neighbouring
  points ?
• Using evidential approach allows one to
  calculate these values using the reliability
  parameters of the sensor.

15
More on Evidential Theory

• Operates with both occupancy and emptiness
  evidence m(o) and m(e), which are initially set
  to zero.
• Accumulates them independently.
• Provides Belief and Plausibility measures
• Bel m(o) Pl
  1 m(e)
• Advantages
• Allows to use sensor error analysis in
  calculating the evidence of range data.
• Resolves "contradictory vs unknown" ambiguity
  (when m0.5), by providing an interval of
  uncertainty BelPl.

16
Sonar Sensor model

• Probabilistic approach
• P(occ) 1 P(emp)
• Evidential approach
• On the arc
  m(occ)1/N, m(emp)0
• In sector m(occ)0,
  m(emp)1/S
• Elsewhere m(occ)0,
  m(emp)0

17
Visual Sensor model

• Where uncertainty comes from ?
• From the limitations of the camera
• From the complexity of the environment
• What are parameters that determine the
  reliability of data ?
• Feature match error
• Depth calculation error

18
Single Camera Sensor Model
Real camera
Ideal camera
19
Stereo Error Analysis

• Quantization error gives

• Match error E gives

20
Image,Features Range Data
(The brighter a range point, the higher its
evidence)
21
Combining Evidence

• Fusion Problem
• - According to one measurement point A is
  non-occupied
• - According to another point A is most likely
  occupied
• So, is the point occupied or not ?
• How sure are we in the answer ?
• General situation
• Measurements describe not the same point, but
  neighboring points.

22
Conventional Approaches

• Probabilistic approach
• Bayesian rule (requires knowledge of conditional
  probability tables)
• Evidential approach
• Dempster-Shafer rule (assumes DS-independence)

23
Assumption of independence

• In practice simplified rules are used (ULaval)

These rules however cannot be applied for
dependent range data!
Example (from real experiment) The same
unreliable feature is observed twice by the
cameram1 0.6 m2 0.6 result in
m12 0.7

• Other combination rules need to be investigated!

24
Fusion as Regression

• Given a set of sample points
  from a sensor,
• find a function such that
  minimizes the error
• Offers a rule for combining dependent and
  adjacent data
• Requires evidential approach (probability axioms
  no longer hold)
• Questions
• How to incorporate sensor model?
• Which regression technique to use?

25
Incorporating Sensor Model
Each feature induces a set of
features according to the sensor model.

• Use
  it!
• Choose coordinate system - To be convenient for
  1) the application, and 2) sensor model

• Choose constraints - To suit well 1) sensor
  model, and 2) the application

• Generate sample points just enough.

26
Rule-based vs Regression-based

• Conventional approach

Number of points depends on the scale

• Regression based approach approach

3-5 points are enough (scale invariant)
27
Choosing Regression Technique

• Two factors to take into account
• Factor 1 The quality of range data
• Robust regression (LMS, WLS) - deal with outliers
  
• Least Squares estimators fast, but not good on
  outliers

• Factor 2 The use of the constructed model
• Linear functions are easy to invert.
• Linear functions are easy to constrain.

28
Adaptive Logic Network
ALN - Least Squares multiple Linear regression
technique
29
Experiments

• Terms of experiments
• Environment large-scale, full of visual
  features.
• Range data obtained by a single camera sensor.
• Regression ALN-based.

• Goals of experiments to observe the problems
• How are planar objects modeled in a radial-based
  system?
• How much time and memory does it takes to model?
• Is the quality good enough for planning
  navigation?

30
Experiments (Simulated Data)

• Empty rectangular room
• L46

• Cylindrical room with object
• L150
• (Error0.15)

31
Experiments (Real Data)

• The precision of modeling is determined by the
  number of linear pieces used in regression
  L8 L16

• Problems observed
• LS and outliers
• Discontinuity of the models

• Lack of features
• Visualization problem

32
Representing the Occupancy

• Conventional grid representation of mF(x,y,z)
  requires storing and processing huge 3D arrays of
  data
• Solution Parametric representation.
• Any function can be represented using linear
  equations
• Questions
• How to calculate surface equations ? - Use
  regression.
• How to extract navigation maps from them ?

33
Extracting maps
1. Obtain the inverse (which is possible due to
the monotonicity constraint)
2. Apply at different heights h, shrink and take
the intersection

• The obtained polygon can be used in navigation
  planning.

34
Planning Navigation

• Use Reinforcement Learning or Potential Fields
  approach.
• Reinforcement of action a (where to go) taken at
  state S (of the robot and environment) is r(S,a)
  rgoal rexploration robstacles
• Obstacle points repel robstacles lt 0
• Goal and exploration points attract rgoal gt 0,
  rexploration gt 0
• Optimal navigation policy p maximizes the total
  discounted reinforcement defined by the Value
  Function
• Value Function of the optimal policy satisfies
  Bellmans equation

35
Planning Navigation (cntd)

• Obstacle and exploration points are extracted
  from the maps

• Value function is calculated using ALN-based
  reinforcement learning.

36
Putting it together
Explore an environment with a video-camera''
Explore environment (e.g. to locate an object) by
planning intelligently navigation using compact
and convenient models constructed by combining
uncertain dependent range data registered by an
affordable and fast 3D range sensor.
37
Robot Boticelli

• Probabilistic

38
From Image Processing to Navigation planning
Movie 1 From 2D features to 3D depth map to 3D
occupancy models to 2D navigation polygons.
Movie 2 Looking for the target in unknown
environment.
39
Conclusions

• The problems of occupancy modeling identified.
• Evidence-based visual sensor model is developed
• (allows one to use an off-the-shelf
  video-camera).
• Regression-based range data fusion technique is
  developed
• (allows one to combine dependent data).
• Methods for building and using parametric
  occupancy models
• (require little memory and yield efficient map
  extracting).
• Framework established. Major problems identified.
• Further improvements shown.

40
Directions for Further Improvement

• Better range data
• Better cameras, better calibration, image
  rectification
• Better feature selections, robust tracking
• Other range sensors (e.g. laser range, infrared
  scanners)
• Other regression and neuro-computational
  techniques
• Belief Occupancy in addition to Plausibility
  Occupancy
• Combine several models into one model
• Work on improving the accuracy and apply to other
  exploration tasks (e.g. building virtual
  environments, robot localization, tele-robotics,
  texture mapping)

41
Acknowledgements

• This research was partially supported by the
  Defense Research Establishment Suffield.
• Thanks to employees of Dendronic Decisions
  Limited, Edmonton, Canada Kyle Palmer, Monroe
  Thomas, Brant Coghlan and Ron Kube - for hardware
  support and GUI design.
• The influence of Sandro Botticelli and other
  artists of the Renaissance, the cultural movement
  concerned with the multifaceted development of a
  human being, is acknowledged as well.

42
Neurocomputational Approachfor Modeling
Environmentsfrom Uncertain Range Data
Dmitry O. Gorodnichy and William W.
ArmstrongDepartment of Computing
ScienceUniversity of Alberta http//www.cs.ualber
ta.ca/dmitri
43
Outline

• World Modeling
• Why Vision ? Why Occupancy Approach ?
• Problems and alternative solutions
• Designing fast and affordable 3D sensor
• Visual sensor implementation Calculating
  reliability values
• Combining dependent range data
• Representing the occupancy
• Memory consumption Utility to application
• Putting into a framework. Conclusions.