nus.soc.cs5248 Ooi Wei Tsang

1
Intermedia Synchronization

• Ooi Wei Tsang

2
Overview

• Why we need to sync?
• How close must we sync?
• How to sync?
• How to measure sync?
• Applications

3
Synchronization
Intrastream
Interstream
Local playback
Networked
4
Problem

• Ensure that audio/video captured at the same
  instance at the source, are displayed at the same
  instance at the receivers.

5
Why we need Synchronization?
6
Why not Pre-sync?
Sender
Receiver
7
Seperating A/V Flows

• Can receive audio without video
• Can prioritize audio over video

8
May Come from Different Sources

• UNLV Access Grid

V
V
A
A
Sender
Receiver
Sender
Receiver
9
or Displayed at Different Sources
V
A
V
A
Sender
Receiver
Sender
Receiver
10
More Complications

• Different decoding time for video and audio
• Network delay and jitter
• Different clocks

11
Bottom Line

• Must synchronize at the receiver

12
Synchronization Requirements
13
Reference

• Human Perception of Jitter and Media
  Synchronization
• by Ralf Steinmetz
• IEEE JSAC 14(1) 1996

14
Results
Badness
A/V offset
-80ms
80ms
15
What matters

• Video ahead of audio vs. audio ahead of video
• Size of head
• Occupancy

16
What does not matter

• Language
• Speaking or Singing
• Hobby
• Speed

17
Conclusion

• -80ms to 80ms acceptable
• -160ms to 240ms noticable but not annoying
• - audio gt video
• video gt audio

18
How to Synchronize?
19
Receiver needs

• Exact time a sample is generated
• Each packet needs a timestamp

20
RTP Protocol
RTP Hdr
Media Timestamp
21
But..

• Media timestamp has a random offset

22
RTCP Protocol

Media Timestamp
RTCP Hdr
NTP Timestamp

23
NTP

• Network Time Protocol
• Synchronize clock up to few milliseconds

24
Audio RTCP
Audio RTP
Sender Audio
Video RTCP
Video RTP
Sender Video
Receiver Video
Receiver Audio
25
Audio RTCP
Audio RTP
Sender Audio
Video RTCP
Video RTP
Sender Video
Receiver Video
Receiver Audio
Synchronization Delay
26
Multiple Receivers

• Everyone must have the same synchronization delay

27
How to set Sync Delay?

• Receivers tell each other their network delay
• Synchronization Delay max Network Delays
  Buffer

28
Buffer Control
29
No Delay No Jitter
30
Delay, No Jitter
31
Delay, Jitter
32
If Buffer too Full/Empty

• Send message Play faster! or Play slower! to
  other receivers

33
Must Synchronize!

• Send message Play faster at time T! or Play
  slower at time T! to other receivers

34
How do we Measure Synchronization?
35
Reference

• A Method and Apparatus for Measuring Media
  Synchronization
• by Brian Smhmidt etc.
• NOSSDAV 1995

36
System under Test
Generator
Measurement System
37
Beyond Video Conferencing
38
HQ Video and Audio

• Multiple Channels
• Multiple Displays

39
Network Musical Performance
BOS
BOS
SF
SF
BOS
ACM Multimedia 94
40
NASDAQ

• Synchronize display of stock quotes