Motion Analysis of OmniDirectional Video Streams for a Mobile Sentry

1
Motion Analysis of Omni-Directional Video Streams
for a Mobile Sentry

• Tarak Gandhi
• Mohan Trivedi
• Computer Vision and Robotics Research Laboratory
• University of California at San Diego

2
Motivation

• A mobile platform with a camera mounted on it can
  continuously monitor a large area.
• Such a mobile sentry can relieve security
  guards of strenuous duty.
• The system should detect and record interesting
  events such as independently moving objects.

3
Motivation

• Motion is used to discriminate between objects
  and background.
• Since the camera is moving, the background in the
  image undergoes motion.
• Camera vibrations add to this motion further
  complicating estimation.
• This ego-motion should be compensated to avoid
  false alarms due to moving background features.

4
Mobile Platforms
5
Omni-Directional Vision Sensor (ODVS)

• Central Has single viewpoint.
• Panoramic Covers 360 degrees surroundings.
• Catadioptric Contains mirror and lens

6
Why Use ODVS

• A single ODVS camera gives a full panoramic view
  and helps maintain surround awareness.
• Rectilinear cameras with small FOV cause motion
  ambiguities.
• Motion field produced by horizontal translation
  is similar to that of rotation around vertical
  axis.
• Wide FOV in ODVS avoids this problem Gluckman98.

7
Video from a Mobile Platform
8
Previous Work

• Most ODVS motion analysis methods Gluckman98,
  Daniilidis02, Vassalo02, Shakernia03 use full
  optical flow reliable at corners-like points.
• Direct methods Black96,Irani98,Odobez98 use
  gradient information from edges and other
  textured areas to fit parameterized motion model.
  These methods have been applied to rectilinear
  cameras.

9
Proposed Approach

• Formulates direct motion estimation for omni
  cameras using planar motion model.
• Combines motion transform with ODVS transform.
• Compensates ego-motion of ground plane.
• Captures interesting events having independent
  motion or height above ground plane.

10
Optical Flow

• Apparent motion observed in image sequence is
  known as optical flow.
• Under favorable conditions, optical flow
  constraint is satisfied
• One equation, two variables Aperture problem!

11
Aperture Problem

• No motion information in uniform regions.
• Only normal component of motion available on edge
  points.
• Corner points give full motion.

12
Direct Motion Estimation

• Planar motion model
• Substitute this in the optical flow equation
• Solve using linear least squares.
• To generalize for omni camera, combine motion
  model with omni image transforms.

13
Hierarchical Motion Estimation

• Optical flow constraint valid only for small
  motion.
• Iterative coarse to fine (pyramid) method is used
  to account for larger motion.

14
Hierarchical Motion Estimation
15
System Block Diagram
GPS navigation, Camera calibration
Video stream
Compute warping transform
Delay
Warp
Space, Time gradients
Bayesian motion parameter estimation
Normalized image difference
Post processing, clustering, tracking
Event database
16
Planar Motion Model

• Motion of planar surface is modeled using
  projective transform.
• The homogenous coordinates of the perspective
  transforms of a point on planar surface in two
  images are related by

17
ODVS Transform

• The transformation between the perspective and
  ODVS image coordinates to correct for the
  distortion.
• Direct warping using this transform can degrade
  image quality in outer areas.

18
ODVS Transform
19
Motion Compensation in Omni Domain
wa(ua,va)
pa(xa,ya)
Pa(Xa,Ya,Za)
K-1
F-1
H
F
K
wb(ub,vb)
Pb(Xb,Yb,Zb)
pb(xb,yb)
20
Bayesian Estimation
Motion parameters
Planar motion model
ODVS transform
Optical flow constraint
State vector
Measurement equations
Functions ci(x) and Jacobians Ci are given by
composition of individual transforms.
21
Bayesian Estimation

• Estimation model
• Use Iterated Extended Kalman filter measurement
  update equations
• Priors obtained from approx. calibration, speed.

22
Combining Motion and ODVS Transforms

Planar motion
ODVS transform
Calibration
Optical flow constraint
23
Outlier Removal

• Estimation can be adversely affected by presence
  of outliers.
• To remove outliers, the error residual in each
  sample is compared with expected covariance.
• Expected covariance given by

24
Motion Estimation Algorithm
Form Gaussian Pyramids from images A, B
Initialize x ? x- , P ? P-
For Level N to 1 Perform multiple iterations
Finish
Warp B using x
Obtain gx, gy, gt using A and warp(B)
Use residuals to classify inliers/outliers
Update x, P using measurement equations on inliers
25
Post-Processing and Tracking

• After motion compensation, features on road are
  aligned, obstacles misaligned.
• Normalized difference between consecutive frames
  used to detect obstacles.
• Nearby blobs clustered together and tracked over
  frames.

26
Event Detection
Estimated parametric motion
Features Black unused,Gray inliers, White
outliers
Normalized difference image
Detected objects
27
Event Detection Video
Estimated parametric motion
Features Black unused,Gray inliers, White
outliers
Normalized difference image
Detected objects
28
Interesting Events
FA
FA
FA
FA
FA False Alarm
29
Event Mapping
Approximate event locations
Sentry path
30
Event Record for Database
31
Performance Evaluation
32
Event Detection
Estimated parametric motion
Features Black unused,Gray inliers, White
outliers
Normalized difference image
Detected objects
33
Automobile Surround Analysis
Estimated parametric motion
Features Black unused,Gray inliers, White
outliers
Normalized difference image
Detected objects
34
Contributions

• Formulated direct motion estimation approach for
  ODVS cameras by combining planar motion model
  with ODVS transform.
• An iterative coarse-to-fine Bayesian framework
  used for optimal fusion of prior knowledge with
  image information.
• Detected events such as moving persons and
  automobiles in video sequences from mobile
  platforms.

35
Acknowledgements

• Grant from Technical Support Working Group (TSWG)
  of US DoD.
• Members of CVRR laboratory who helped in
  experiments.

36
Thanks!