Augmented%20Virtual%20Environments%20(AVE):%20Dynamic%20Event%20Visualization

1
Augmented Virtual Environments (AVE) Dynamic
Event Visualization

• Ulrich Neumann Suya You
• Integrated Media Systems Center
• University of Southern California
• September 2003

2
Problem Statement

• Imagine dozens of video/data streams from people,
  UAVs, and robot sensors distributed and moving
  through a scene
• Problem visualization as separate
  streams/images provides no integration of
  information, no high-level scene comprehension,
  and obstructs collaboration

3
A Simple Example USC Campus
1
2
Visualization as separate streams provides no
integration of information, no high-level scene
comprehension, and obstructs collaboration
3
4
AVE Fusion of 2D Video 3D Model

• VE captures only a snapshot of the real world,
  therefore lacks any representation of dynamic
  events and activities occurring in the scene
• AVE Approach uses sensor models and 3D models
  of the scene to integrate dynamic video/image
  data from different sources

• Visualize all data in a single context to
  maximize collaboration and comprehension of the
  big-picture
• Address dynamic visualization and change
  detection

5
Research Highlights Progress
Algorithm research and tech barriers inherent in
AVE system

• Interactive Modeling System
• semi-automated feature finding
• linear/non-linear element fitting
• ARMY TEC tech-transfer over summer 04
• ICT tech-transfer over summer 04 and extension
  for rapid modeling underway
• Video Capture System
• up to 8 real-time internet video streams at
  704x480 at 12Hz with graceful degradation
• Rendering System
• real-time GPU code on dual CPU PC
• integrated camera calibration system
• texture retention and background modeling
• interactive GUI and remote control for
  integration with Northrop Grumman
• Image Analysis System
• detection and tracking of moving objects (people
  and vehicles)
• dynamic creation and placement of pseudo-models
  in 3D scene
• rapid interactive modeling of object models

6
Integrated Modeling System based on
interactive fitting
7
Video Capture System

• AxisCam 2120 (and similar) 704x480 image size
• Three potential bottlenecks
• (i) Ethernet bandwidth (gt20 cameras for 100bT)
• 43 Kb per image at high quality compression
• 12 fps camera image rates
• bandwidth per camera is 12438 KB/s 4.13 MB/s.
  
• (ii) MJPEG decompress time (8-10 cameras --
  current limit)
• 100 fps per CPU with Intel IP library (MMX, 3D)
  for IDCT
• 8 streams _at_ 12 fps consumes one CPU
• MPEG-2 cameras are coming (expect gt20 cameras)
• (iii) Graphics system bus bandwidth (gt150
  cameras!)
• AGP 8x and PCI-Express are fast

8
Rendering and Application Control

• High performance rendering with real-time GPU
  code on dual CPU PC
• Network video and XML interface for integration
  with existing sensor networks and monitoring
  systems
• Views automatically zoom to alarms,
  geo-referenced positions, or user selected views
• Alarm status icons reflect site status
• Patrol-mode automatically flies user-defined
  path(s) over the entire site
• Integrated camera calibration system
• Local and/or remote user(s) control system via
  joystick, keyboard, and mouse
• Customizable control and interface modules and
  SDK in development

9
Video Processing
Dynamic Event Analysis

• Dynamic object detection
• Background estimation
• A variable-length temporal averaging algorithm
• Adaptive histogram thresholding
• foreground object segmentation
• Morphological filtering noise rejection
• Object tracking
• Optimal matching between current objects and
  history
• Consider both spatial and temporal coherence
• Matching criterions overlap, size, motion

10
Dynamic Modeling

• Object modeling
• dynamic polygon model to fit segmented
  silhouette
• planar or 3D generic models
• Object placement
• 3D parameters (position, orientation and size)
  based on
• ground assumption
• object class constraints (e.g. vehicles have 4
  wheels)
• path smoothing

11
Interactive Rapid Modeling (initial
activities)
Modeling from single (and multiple images) by
fitting to parameterized base models What can be
done in a minute?
3
2
1
12
Tracking and Modeling Results
13
Video Texture Management

• Problem Video texture projection paints the
  scene each frame
• Object movement changes visibility/occlusion
• Camera movement paints new areas
• Dynamic visualization control (viewpoint, image
  inclusion, blending, and projection parameters)
  reveals unseen areas
• Texture management provides more complete view
  of the scene
• Texture retention shows recently seen data
• currently occluded by moving object
• currently unseen by moving camera

14
Video Texture Retention

• Challenges
• Accumulation of video textures leads to infinite
  texture storage and system slowdown
• Merging of multiple textures requires accurate 3D
  sensor calibration
• Varied image-to-object mapping between frames,
  illumination, resolution etc
• A Model Based Approach
• a base-texture-buffer for groups of model
  polygons
• warp each new image to the base-buffer via sensor
  model and 3D model
• visibility/occlusion test via depth map
• rendering by traditional texture mapping with the
  base textures - fast

15
Texture Refinement

• Artifacts arise in reconstructed textures from
  tracking errors
• Use 2D image registration in base-texture buffer
• affine motion model for image registration
• feature (corners) based matching
• least squares solution for warp-parameter
  optimization

before
after
16
Texture Management Results

no texture management
with texture management
17
Northrop Application Collaboration

• Military base surveillance situational
  awareness
• Different sensors (CCD/IR cameras, motion
  detectors, radars) deployed throughout the test
  site
• Networking and XML interface communicate with the
  sensor network and a main control station
• System monitors and responds to sensor status and
  alarms, providing overall site visualization
• Views automatically zoom to alarms,
  geo-referenced pre-sets, or user controlled views
• Patrol mode automatically flies user-defined
  path(s) over the entire site

18
Surveillance Scenario
19
Future Plan

• Tracking of PTZ Cameras real-time tracking
  using model-based vision and calibrated mounts
• Dynamic Modeling real-time tracking and
  model-fitting for moving objects
• External, in-line processing
• Texture Management real time texture retention,
  progressive refinement
• System Architecture - scalable video streams and
  rendering capability, (PC clusters?)

20
Interactions

• Collaboration with Northrop Grumman
• Install v.1 system (8/03) for demonstrations
• Install v.2 system (9/04) for demonstrations and
  evaluation license
• Tech transfer
• Source code for LiDAR modeling to ARMY TEC labs
• LiDAR modeling and AVE visualization integration
  into ICT training applications for MOUT
  after-action review
• Publications
• IEEE CGA Approaches to Large-Scale Urban
  Modeling
• PRESENCE Visualizing Reality in an Augmented
  Virtual Environment
• IEEE CGA Augmented Virtual Environments for
  Visualization of Dynamic Imagery
• CGGM03 Urban Site Modeling From LiDAR
• VR2003 Augmented Virtual Environments (AVE)
  Dynamic Fusion of Imagery and 3D Models
• SIGMM03 3D Video Surveillance with Augmented
  Virtual Environments
• Demos/proposals/talks
• NGA, NRO, ICT, Northrop Grumman , Lockheed
  Martin, HRL/DARPA, Olympus, Airborne1, Boeing