DYNAMIC OBJECTS MODELING AND 3D VISUALIZATION

1
DYNAMIC OBJECTS MODELING AND 3D VISUALIZATION

• Ismail Oner Sebe
• Suya You, Ulrich Neumann
• Integrated Media Systems Center
• University of Southern California

2
Motivation

• Analysis, Integration, and Visualization of
  multiple sources for wide area monitoring
• Video Cameras
• Aerial Images
• LIDAR Data (3D models)

Augmented 3D World
3
Motivation

• Current Surveillance Monitoring Center
• Overwhelmed with data fusion and comprehension of
  multiple image streams.
• Limited number of displays
• Waste of Resources

• Better Surveillance System
• Better understanding of streams
• Better use of resources
• Additional capabilities tracking, statistics

USC Public Security Surveillance Center
4
Outline

• Augmented Virtual Environment (AVE)
• Dynamic AVE
• Dynamic Object Detection
• Dynamic Object Modeling
• Dynamic Object Visualization
• Conclusion and Future work

5
AVE vs. Others

• Augmented Virtual Environment
• Fusion of dynamic imagery with 3D models in a
  real-time display to help observers comprehend
  multiple streams of temporal data and imagery
  from arbitrary views of the scene Neumann03
• Related Work
• Distributed Interactive Video Array (DIVA) at
  UCSD Hall02
• VideoFlashlight at Sarnoff Corporation Kumar00
  
• Video Surveillance and Monitoring (VSAM) at CMU
  Kanade98
• Virtual Soccer Match at Keio University, Japan
  Inamoto03

Neumann03 Neumann U., You S., Hu J., Jiang
B., and Lee J. Augmented Virtual Environments
(AVE) Dynamic Fusion of Imagery and 3D Models
VR03, March 2003 Hall02 Hall B. Trivedi M.
A novel graphical interface and context aware
map for incident detection and monitoring, 9th
World Congress on Intelligent Transport Systems,
October, 2002. Kumar00 Kumar R. Sawhney H.S.
Guo Y. Hsu S. Samarasekera . 3D manipulation of
motion imagery, ICIP2000, September 2000
Kanade98 T. Kanade, R. Collins, A. Lipton, P.
Burt and L. Wixson, Advances in cooperative
multi-sensor video surveillance, Proc. of DARPA
Image Understanding Workshop, Vol. 1, pp. 3-24,
1998 Inamoto03 Inamoto,N.,H. Saito (2003).
Immersive Observation of Virtualized Soccer Match
at Real Stadium Model. ISMAR 2003. pp 188-197
6
AVE System Components

• Accurate 3D models
• Scene model as substrate
• Accurate 3D sensor models
• Sensor calibration tracking
• Video analysis
• Dynamic object detection
• Dynamic object modeling
• Dynamic object tracking
• Dynamic visualization
• Data fusion video projection

7
Dynamic Objects Detection

• Background subtraction-based segmentation
• Background Estimation is an variable length
  time-averaging algorithm (Special implementation
  without large buffer requirement)
• Histogram-based Thresholding labels only top 95
  or 25/255 (whichever is larger) of intensities as
  foreground
• Morphological filtering acts as an noise cleaning
  Fast Image Closing
• Segmentation is a two-pass 4-neigbor connectivity
  algorithm

8
Dynamic Objects Modeling I

• Dynamic Objects are modeled using several
  properties extracted from video
• Location The center of the object
• Size Estimated from the enclosing box
• Motion Vectors Estimated using arbitrary shape
  translational rigid-body movement
• Similar to MPEG motion vectors for arbitrary
  shapes
• One motion vector per each object is estimated
• Shape Convex Hull of the object boundary

9
Dynamic Shape Modeling II

• Why Convex Hull?
• Background segmentation quality is low in case of
  camouflage (foreground and background having same
  color information)
• These spurious splits and holes can be filled by
  using convex hull approximation of the segmented
  object
• On the other hand, background may leak into
  foreground object in case of non-convex objects
• A fast (O(N log N) complexity) model fitting
  convex hull method Andrew, 1979
• Convex hull performs well in terms of getting the
  general outline of the objects
• Prone to over-convexation, e.g. shadows

10
Dynamic Shape Modeling III

• 2D convex-hull representation to model the moving
  objects

• Segmented video will be used in both tracking and
  visualization of the dynamic objects in the scene
  

11
Dynamic Objects Tracking I

• How to combine all object properties in one
  general framework
• Match Criterion Overlap, Size match, Motion
  Match

• Matching of two scale numbers is made by the
  ratio of geometric mean and the arithmetic mean,
  i.e.
• Ratio1, when AB
• Ratio0, when AgtgtB or BgtgtA or B0 or A0

12
Dynamic Objects Tracking II

• Matches are calculated and stored in a NxM
  resemblence matrix (R)
• N new objects
• M previously tracked history objects
• 3 Modes Appear, Disappear, Track
• Assignments by Winner-Takes-All Algorithm
  (Iterative Greedy Search)

Do Find Max(R) If Max(R) gt Threshold
Track , Remove that row and column Else
New object Appears, Remove that row Until
Objects Remain Remaining History objects Disappear
13
Dynamic Objects Tracking III

• An Example
• 2 new objects, N2
• 3 history objects, M3

Do Find Max(R) If Max(R) gt Threshold
Track , Remove that row and column Else
New object Appears, Remove that row Until
Objects Remain Remaining History objects Disappear
R
Decisions

• 1-gtC

• 2-gtA

• B Dissapear

14
Dynamic Objects Visualization

• Dynamic objects are visualized with a similar
  technique to spirits
• Dynamic objects are assumed to be on the ground
• Their 3D position is estimated by intersecting
  with the ground plane
• n -v (objects are assumed to face the camera)
  

A Camera Center B Bottom-Middle of Bounding
Box C 3D point of the object v Cameras
Normal n Objects Normal
15
Screen Shots

• Without any prior knowledge of the objects
  under investigation
• Views are arbitrary, and different from the
  actual camera location

16
Conclusion

• A novel visualization system for video
  surveillance based on tracking and 3D display of
  moving objects in an Augmented Virtual
  Environment (AVE) is presented
• Adaptive background-subtraction method combined
  with a pseudo-tracking algorithm for dynamic
  object detection
• Visualization of the dynamic objects in the AVE
  system
• Fusion of all the video, image and 3D models

17
Acknowledgement

• Integrated Media Systems Center, USC
• NIMA
• MURI team Avideh Zakhor (UC Berkeley), Suresh
  Lodha (UC Santa Cruz), Bill Ribarsky (Georgia
  Tech), and Pramod Varshney (Syracuse)