Digital Amphitheatre

1
Digital Amphitheatre

• Allison Mankin Colin PerkinsRon Riley Ladan
  Gharai
• Information Sciences InstituteUniversity of
  Southern California
• Next Generation Internet Program, DARPA ITO

2
Aims of the digital amphitheatre

• Facilitate a meeting such as this, through a
  networked teleconference
• Combine audience, panel and speaker into a single
  seamless display
• a virtual auditorium or meeting room setting
• up to hundreds of audience members
• retain a sense of presence
• add information structure (as PDAs may do in a
  live meeting)
• Security and manageability

3
Challenges

• Scaling to large interactive sessions
• Developing architecture and control protocols for
  large scale network conferencing
• Enhancing shared presence
• Background substitution
• Overcoming network limitations
• With SuperNet and G2D, bandwidth is now available
• Overcoming end-system limitations
• NIC, bus, interrupts, packet processing, protocol
  processing, OS, application, rendering, display

4
Challenges

• Innovative architectures
• Distribute application processing through the
  network
• Make innovative use of Internet protocols (SDP,
  SIP)
• Augmented meeting experience
• Integrated information and environment

5
Concept

• Mel

6
Implementation
7
Key components
Background substitution
User interface
8
User interface

• Audience members are given a common background
  and seats
• User software performs background removal and
  substitution algorithm before sending
• Audience members are given naturalistic seating
  (no re-shuffling, empty chair on departures, may
  eventually be refilled)
• Distributed seating algorithm collaborating
  video mergers

9
User interface

• Accommodate large numbers on desktops
• Exploit good PC capabilities
• Information rich, scaleable, yet uncluttered
• Participant information and network statistics
  available, yet hidden by default
• Flexible
• Use on multiple desktop displays
• Use in hybrid with live meeting room

10
Building the user interface
Panelists
Speaker or Room
B2
B2
C1
C1
A3
B5
A3
Merged Audience
B5
A1
A2
A3
A4
A5
B1
B2
B3
B4
B5
C1
C2
C3
C4
C5
A1
A2
A3
A4
A5
B1
B2
B3
B4
B5
C1
C2
C3
C4
C5
Conference GUI
11
Video merge agents

• To enable the numbers of images displayed by the
  receivers, agents aggregate video throughout the
  network
• Distributed algorithm to decide seating
  arrangements and pass correct background image to
  senders
• Also handle joins/leaves and authentication of
  new receivers

12
Status of merge agents

• Preliminary implementation
• Merge up to 15 streams into one and show many
  merged streams
• Standalone application
• Enhancements
• Merging more streams (effects on RTP?)
• Investigating a range of codecs/transcoding
• YUVCR, Intra-H.261, DV
• More coordination effects

13
Background substitution

• Performed locally by each sender
• Background to be substituted is distributed by
  the session control protocol, giving a seamless
  image
• Real-time algorithm, with low processing cost on
  full-motion capture
• Preliminary implementation is demonstrated today
• Will be open-source when completed

14
Background substitution

• Capture an image of the background
• Add the subject (user waits briefly before
  entering)
• Difference the frames, giving a mask
• Overlay with the conference background
• Replace pixels in image based on the mask

15
Background substitution

• Capture an image of the background
• Add the subject (user waits briefly before
  entering)
• Difference the frames, giving a mask
• Overlay with the conference background
• Replace pixels in image based on the mask

16
Background substitution

• Capture an image of the background
• Add the subject (user waits briefly before
  entering)
• Difference the frames, giving a mask
• Overlay with the conference background
• Replace pixels in image based on the mask

17
Background substitution

• Capture an image of the background
• Add the subject (user waits briefly before
  entering)
• Difference the frames, giving a mask
• Overlay with the conference background
• Replace pixels in image based on the mask

18
Background substitution

• Capture an image of the background
• Add the subject (user waits briefly before
  entering)
• Difference the frames, giving a mask
• Overlay with the conference background
• Replace pixels in image based on the mask

19
Control and Protocols

• The video merging and background substitution
  need distributed control
• to locate merge agents and handle joining
• to ensure the correct background is substituted
• And use of Internet protocols in novel ways
• SIP to initiate and control sessions
• SDP to describe seating arrangements
• Mbus for co-ordination among media (a/v,
  whiteboard)

20
Integrated Robust-Audio Tool
21
Integration of shared workspace

• User-interface challenge
• Real-estate. But two displays on desk are
  increasingly common, especially LCDs
• Synchronization and coordination among
  applications and displays
• Message bus infrastructure
• Integration via middleware (Mbus, CORBA)
• Future work...

22
Schedule for digital amphitheatre

• Design paper was published in June 2000
• Second demonstration today
• Desired next steps
• Test with hundreds of live streams (vs.
  prerecorded)
• Hold meeting in DA
• Use to enhance a sizeable collaboration
• Standalone video tool (vic) with background
  substitution added - release in November
• Full release in summer 2001 (2nd year)

23
Applications

• Meetings
• Allow more frequent interactions, familiarization
  among multiple projects members
• Allow introduction of information enhancements
  into ordinary speaker-centered conferences
• Training
• Establish presence of the remote participants to
  speaker/commander/instructor
• Lower barriers for new participants to begin
  relation to the group?
• Hybrid meetings

24
Summary

• Introduction to the digital amphitheatre
• Application ideas
• Demonstration

25
Video merge agents
The video merger is an RTP mixer combining
multiple video streams into an audience segment
1
VM
2
1
2
N
Merged audience sections ? Many fewer
packets/frames/streams ? Less data, decreased
packet interrupt processing, rendering...
N
26
Video merge agents
Multiple mergers send to the user interfaces,
which composite these segments forming the
audience
Site B
VM
VM
Site A
VM
27
Integrated Robust-Audio Tool

• Error protection
• FEC, interleaving
• Concealment
• Layered coding
• Network adaptation
• Clock skew
• Jitter
• Transcoder/mixer

• Range of codecs
• 8kHz - 48kHz
• Mono/stereo
• 3d audio
• Message bus
• remote control
• lip sync
• integration with video tool
• Security