Streaming of Panoramas

1
Streaming of Panoramas
Treffen der ITG FG 3.1.2 Sci-Worx, Hannover, 2002

• Dr.-Ing. Peter Eisert
• Computer Vision Graphics
• Image Processing Department
• Heinrich-Hertz-Institute, Berlin

2
Motivation
3D Streaming / Mobile Rendering

• Interactive Navigation in Virtual Worlds on
  Mobile Devices
• Virtual city guides (position dependent)
• Tourism applications
• E-commerce
• Requirements / Constraints
• 3D scene description located on a
  server(cylindrical panoramas, concentric
  mosaics, light fields)
• Interactive navigation with low
  delay(Transmission of navigation decision /
  position of user)
• Low data-rate of wireless channels
• Restricted computational power and storage of PDAs

3
Overview

• Image-based rendering
• Cylindrical panoramas
• Concentric mosaics
• Light fields
• Streaming of image-based datasets
• First implementation
• MPEG-4 client/server system
• Streaming of high resolution cylindrical
  panoramas
• Demo

4
Image-based Rendering
Real Picture
Computer Graphics

• Simple acquisition
• Fast rendering
• Natural impression

• Modeling time consuming
• Powerful rendering necessary
• Artificial appearance

5
Cylindrical Panoramas

• Simple acquisition
• Rotating camera or mirror system
• Vertical field of view limited
• Navigation Rotation and zoom
• QuickTimeVR

6
Concentric Mosaics

• 3D plenoptic function
• Navigation Rotation, translation in a plane and
  zoom
• Camera motion constrained on planar concentric
  circles
• Acquisition of n1 vertical lines (slit
  images)at different radial positions?
  horizontal parallax
• Each pushbroom camera contributes one cylindrical
  panorama

7
Example of a Concentric Mosaic
a)
b)
c)

• a) Panorama acquired at the rotation center (CM0)
• b) Panorama acquired at the outmost position
  (CMn)
• c) Panorama as in b) but looking in the opposite
  direction

8
Rendering of new Views II
9
Simplified Acquisition

• One single off-center camera rotating on an arm
  with radius Rn
• a) Free movement inside a circle of radius
• b) Free movement inside the ring

Normal Viewing Direction
Tangential Viewing Direction
10
Demo Concentric Mosaics (Zhang, Li)
11
Light Fields

• Static 3D scene
• Known viewing positions
• 2D array of images
• 4D plenoptic function(4D array of rays)
• no time dependency
• no wavelength
• no scattering in air (bounding box)
• Rendering
• arbitrary viewing position and direction
• new view interpolated from stored images
• no geometry required

12
Light Field Representation
ST array of UV images
UV array of ST images
13
Interpolation

• 8 x 8 images of resolution 256 x 256
• Aliasing!!
• Many images needed ? large data sets

14
Example Dragon (Stanford)

• 4 slabs
• 24 x 24 images each slab
• resolution 256 x 256 pixels

15
3D Streaming

• System for streaming image-based data
• Virtual walk-throughs
• E-commerce
• Free navigation in 3D scenes
• Visualization on portable devices
• Limited band-width and storage
• Efficient coding and transmission necessary

16
Interaction of all Components
Photorealistic Visualization

• Image-based representation
• Large amount of data

Free navigation
Representation

• Fast transmission of requested data
• Prefetching necessary (RTTgt100ms)
• Intelligent selection of transmitted data

• Efficient coding necessary
• Scalability
• Adapted to representation

Navigation
Coding
Streaming

• Efficient packet-based protokoll
• RD optimization with delay constraints

Bit-Rates 100 kbit/s - 2 Mbit/s
17
Current Implementation

• Cylindrical, high resolution panoramas
• MPEG-4 BIFS scene

Representation

• JPEG
• JPEG-2000

Navigation
Coding

• MPEG-4 BIFS Player

Streaming

• MPEG-4 client/server architecture
• TCP / UDP

18
Cylindrical Panoramas

• Navigation Rotation / Zoom
• Random access to image patches
• Pre-fetching realized by visibility sensors

19
Visibility Sensors

• Pre-fetching of neighboring patches
• Removal of patches that are no longer used

20
Demo Cylindrical Panorama
21
Summary

• Image-based rendering provides tools for
  photo-realistic visualization of 3D scenes
• Cylindrical panoramas / concentric mosaics /
  light fields
• Large datasets require efficient encoding and
  transmission
• Interactive navigation on mobile devices
• New applications
• virtual guides
• e-commerce
• MPEG-4 client/server system for streaming high
  resolution panoramas

22
Thank you