Internet AV Applications

1
Internet A/V Applications
2
Real Time Delivery interactive A/V

• packet network nature
• delay, unpredictable delay
• loss
• consequences ? countermeasure
• pkts arrive out of order --gt reordering
• pkts arrive out of sync --gt jitter buffer
• pkts lost --gt acknowledgement and retransmission
• no-no for real time delivery
• mechanism
• order sequence number used in packet handling
• 1 each packet --gt gaps loss
• time timestamp used in playback
• encoding time, arrival time and playback time
• can also be used for delay jitter calculation

3
Interactive Audio/Video

• Sending Side
• A sequence number on each packet is required
• A encoding timestamp is required on each packet
• Even if there is no packets transmitted (e.g.
  silence), encoding timestamp is always ticking
• Same timestamp may appear on multiple packets
• Receiving Side
• A playback buffer is required for jitter
  processing
• In general, packet receiving time is different
  from packet playback time (delayed playback)
• Packet playback time is based on the encoding
  timestamp
• If packet is not available during playback,
  concealment methods are used to fill the gap
• Real-time traffic needs the support of
  multicasting.
• Translation means changing the encoding of a
  payload to a lower quality to match the bandwidth
  of the receiving network.
• Mixing means combining several streams of traffic
  into one stream (conferencing)
• Protocol
• TCP, with all its sophistication, is not
  suitable.
• UDP is more suitable. However, we need the
  services of RTP, another transport layer
  protocol, to make up for the deficiencies of UDP.
• Real-time Transport Protocol (RTP) is the
  protocol designed to handle real-time traffic on
  the Internet. RTP does not have a delivery
  mechanism it must be used with UDP.

4
RFC 1889 (RTP, RTCP) pair
RTCP
5
RTP Packet Header
RFC 1890 audio-video profile
padding (for fixed size block), last byte of pkt
is the pad count
ver 2
contributor
granularity determined by payload type
1 first pkt of a talkspurt, after a silence
period
if this RTP stream is mixed
6
RTP Payload Type

• RFC 1890 audio-video profile
• RTP carries the actual digitally encoded voice
• IP UDP RTP voice/video_payload
• Dynamic payload type (type numbers 96 to 127)
• Support new coding scheme in the future
• The encoding name is significant
• Unambiguously refer to a particular payload
  specification
• Should be registered with the IANA
• SIP SDP can negotiate with names
• RED, Redundant payload type
• Current voice samples previous voice samples
• May use different encoding schemes (more
  bandwidth-efficient)

7
Timestamp, SSRC, CSRC

• Timestamp encoding time
• The clock freq depends on the encoding
• E.g., 8000Hz
• Support silence suppression
• If no packets are sent during periods of silence,
  the next RTP packet may have a timestamp
  significantly greater than the previous RTP
  packet.
• The initial timestamp is a random number chosen
  by the sending application.
• Synchronization Source (SSRC)
• 32-bit identifier
• The entity setting the sequence number and
  timestamp
• Normally the sender of the RTP packet
• Chosen randomly, independent of the network
  address
• Meant to be globally unique within a session
• May be a sender or a mixer
• Contributing Source (CSRC)
• Used to identify the original sources of media
  behind the mixer
• 0-15 CSRC entries

8
Mixers and Translators

• Mixers
• Enable multiple media streams from different
  sources to be combined into a single stream
• If the capacity or bandwidth of a participant is
  limited
• An audio conference
• The SSRC is the mixer
• More than one CSRC values
• Translators
• Manage communications between entities that does
  not support the same coding scheme
• The SSRC is the participant, not the translator.

9
Mixer and Translator example
10
RTCP (RTP Control Protocol)

• A companion control protocol of RTP
• Separate data (RTP) and control (RTCP)
• RTCP allows endpoints to periodically exchange
  control information
• For quality-related feedback and other
  meta-information
• A third party can also monitor session quality
  and detect network problems.
• Using RTCP and IP multicast
• RTP Port typically even-numbered UDP port, not
  fixed port RTCP port RTP port (even) 1
• default RTP/RTCP use UDP port 5004/5005
• typical UDP size limited to few hundred bytes
  (OS, network, fragmentation)
• typical one media (audio, video, . . . ) per
  port pair
• RTCP is optional

11
RTCP Types

• sender report (SR)
• time info for each SSID and bytes send gt
  estimate rate
• timestamp gt synchronization among multiple RTP
  streams (e.g. audio/video)
• reception reports (RR)
• report number of packets received and expected gt
  loss, jitter, round-trip delay
• source description (SDES)
• name, email, location, application,. . CNAME
  (canonical name user_at_host) identifies user
  across media
• Must contain a canonical name (CNAME)
• Separate from SSRC which might change
• When both audio and video streams were being
  transmitted, the two streams would have
• different SSRCs
• the same CNAME for synchronized play-out
• explicit leave (BYE) who leaves session, in
  addition to time-out
• extensions (APP) application-specific (none yet)

12
RTCP packets are stackable

• multiple RTCP packets are combined in a single
  UDP packet
• multiple RTCP report records are combined in a
  single RTCP packet
• SRs and RRs should be sent as often as possible
  to allow better statistical resolution
• New receivers in a session must receive CNAME
  very quickly to allow a correlation between media
  sources and the received media.
• Every RTCP packet must contain a report packet
  (SR/RR) and an SDES packet
• Even if no data to report

13
Streams Synchronization

• RTCP requires each sender to transmit info for
  each active stream
• Sender transmits SDES
• CNAME in SDES is used by receiver to correlate
  different streams
• who is talking?
• Sender transmits periodic SR
• SR contains absolute time value in NTP (Network
  Time Protocol) format and RTP timestamp
• NTP time - elapsed in seconds since 0000,
  1/1/1900 (GMT), 64 bits (200 pico-sec resolution)
• Receiver uses the info to correlate one stream ts
  to wall clock
• Synchronization of multiple streams is then
  possible
• SR also contains QoS feedback for bandwidth
  control

14
Report blocks in SR

• SSRC_n
• The source identifier of the session participant
  to which the data in this RR block pertains.
• Fraction lost
• Fraction of packets lost since the last report
  issued by this participant
• By examining the sequence numbers in the RTP
  header
• Cumulative number of packets lost
• Since the beginning of the RTP session
• Extended highest sequence number received
• The sequence number of the last RTP packet
  received
• 16 lsb, the last sequence number
• 16 msb, the number of sequence number cycles
• Inter-arrival jitter
• An estimate of the variance in RTP packet arrival
• Last SR Timestamp (LSR)
• The last SR received from the source
• Used to check if the last SR has been received
• Delay Since Last SR (DLSR)
• The duration in units of 1/65,536 seconds
• Between the reception of the last sender report
  from the source and the issuance of this receiver
  report block

15
RTP Sessions (1-1, 1-M, M-1, M-M)
player
synchronization
Audio Stream RTP/RTCP
Video Stream RTP/RTCP
Two RTP streams, different timestamp, synchronized
playback
audio session
video session
16
RTP Sessions (1-1, 1-M, M-1, M-M)
mixer
translator
player
synchronization
Audio 1
Audio 2
Audio 3
Audio out
steams demux
audio conf session
audio conf session
17
RTCP Receiver Report

• RR
• Issued by a participant who receives RTP packets
  but does not send, or has not yet sent
• Is almost identical to an SR
• PT 201
• No sender information

18
RTCP Source Description Packet

• Provides identification and information regarding
  session participants
• Must exist in every RTCP compound packet
• Header
• V, P, SC, PT202, Length
• Zero or more chunks of information
• An SSRC or CSRC value
• One or more identifiers and pieces of information
• A unique CNAME (user_at_host) does not change within
  a given session.
• Email address, phone number, name

19
(No Transcript)
20
Calculating Round-Trip Time

• Use SRs and RRs
• E.g.
• Report A-gtB A, T1 ? B, T2
• Report B-gtA B, T3 ? A, T4
• RTT T4-T3T2-T1
• RTT T4-(T3-T2)-T1
• In Report from B
• LSR T1 (time of last SR)
• DLSR T3-T2 (delay since last SR)

B
A
T1
T2
T3
T4
21
Calculating Jitter

• The variation in delay
• The mean deviation of the difference in packet
  spacing at the receiver compared to the packet
  spacing at the sender for a pair of packets
• This value is equivalent to the derivation in
  transit time for a pair of packets.
• Si the RTP timestamp for packet i
• Ri the time of arrival for packet i
• D(i,j) (Rj-Ri)-(Sj-Si) (Rj-Sj) - (Ri-Si)
• The Jitter is calculated continuously
• J(i) J(i-1) ( D(i-1,i) - J(i-1))/16

22
Timing of RTCP Packets

• RTCP provides useful feedback
• Regarding the quality of an RTP session
• Delay, jitter, packet loss
• Be sent as often as possible
• Consume the bandwidth
• Should be fixed to a small fraction (e.g., 5)
• An algorithm, RFC 1889
• Senders are collectively allowed at least 25 of
  the control traffic bandwidth. (CNAME)
• The interval gt 5 seconds
• 0.5 1.5 times the calculated interval
• This helps to avoid unintended synchronization
  where all participants send RTCP packets at the
  same time instant, hence clogging the network.
• A dynamic estimate of the avg. RTCP packet size
  is calculated.
• To automatically adapt to changes in the amount
  of control information carried.

23
Packet Concealment
Missing voice packets and reconstructing the data
stream