Concepts of Multimedia Processing and Transmission

1
Concepts of Multimedia Processing and Transmission

• IT 481, Lecture 8
• Dennis McCaughey, Ph.D.
• 30 October, 2006

2
Broadcast Environment
3
Video Transmission System Example
4
Rate Control

• Video applications involve real-time display of
  the decoded sequence.
• Transmission across a Constant Bit Rate (CBR)
  channel requires a constant end-to-end delay
  between the time that the encoder processes a
  frame and the time at which that same frame is
  available to the decoder
• A buffer is require to match the Variable Bit
  Rate (VBR) encoder with the CBR channel

5
Admission Control

• Decision if a new connection with a given set of
  QoS parameters can be allowed into the network
• Criteria the new connection will not degrade the
  QoS of other ongoing connections
• Need models for predicting the expected bit rates
  of video sources especially packet switched
  networks
• Simpler for circuit switched networks where the
  transmission resources are constant over the
  duration of the connection

6
Usage Parameter Control

• Prevent sources from exceeding the traffic
  parameters negotiated at call setup
• Maliciously or unintentionally
• Example tracking cell rate in an ATM network and
  verifies that the source remains within that rate
• Example Leaky Bucket
• a cell counter incremented with each cell arrival
• Decremented at fixed intervals
• If counter exceeds a threshold cells are dropped
  or marked for deletion

7
Multi-Resolution Encoding

• Separation of the information into two or more
  layers or resolutions
• Coarse Resolution
• Reduce the spatial or temporal resolution of the
  sequence or by having images of lower quality
• Transmitted by high priority packets
• Detailed Resolution
• Transmitted by lower priority packets than can be
  discarded first
• Multi-Resolution Encoding enables efficient error
  concealment techniques
• e.g. interpolate lost information from packets
  that were not lost

8
Layered Video Coding

• Provide multiple image quality levels
  simultaneously across multiple network channels
• e.g. Standard Resolution and High Definition TV
• Each receiver individually tunes its reception
  rate by adjusting the number of layers that it
  receives
• Two problems to solve
• Layered Compression
• Multiple image quality levels
• Layered Transmission Problem
• Selective delivery of layer subsets to individual
  receivers

9
Structure of a Simulcast Coder
10
Simulcast Coder

• Produces a multirate set of signals that are
  independent of one another
• Each layer provides improved quality, independent
  of sublayers
• Single layer (nonscalable) decoder can decode any
  layer
• Total available bandwidth is partitioned
  dependent on the desired quality for each
  independent sublayer

11
Structure of a Layered Coder
12
Layered Compression

• Input signal is compressed into a number of
  discrete layers, arranged in a hierarchy that
  provides progressive refinement
• If first layer is received, decoder will produce
  the lowest quality version of the signal
• If the decoder receives two layers, it will
  combine the second layer with the first layer to
  produce improved quality
• Overall, the quality improves with the number of
  layers that are received and decoded
• Layered compression and layered transmission must
  be combined to be effective

Slide courtesy of Hung Nguyuen
13
Simulcast vs. Layered Approach

• Distortion measures quality degradation between
  reconstructed and original signals
• DI(R) an ideal coder
• DR(R) a real coder
• DL(R) a layered coder
• DS(R) a simulcast coder

14
Layered Transmission

• Each layer is transmitted on a different network
  channel.
• Network only forwards the number of layers that
  each physical link can support.
• Each user receives the best quality signal that
  the network can deliver.
• The network must be able to drop layers
  selectively at each bottleneck link

15
Video Communication System
16
Error Resilience

• Redundancy is added to the compressed bitstream
  to allow the detection and correction of errors
• Can be added in either the source or channel
  encoder
• Shannon Information Theory
• Separately design the source and channel coders
  to achieve error-free transmission so long as the
  source is represented by a rate below the channel
  capacity
• Source should compress the source as much as
  possible
• Channel coder, via Forward Error Correction (FEC)
  adds redundancy bits to enable error detection
  and correction

17
Binary Symmetric Channel
18
Shannons Capacity Theorem

• If the Rate (R) of a code R log2(m)/L is less
  than channel capacity C, there exists a
  combination of source and channel encoders such
  that the source can be communicated over the
  channel with fidelity arbitrarily close to
  perfect
• m Number of message words
• L Number of code word bits

19
Types of Codes

• Block Codes
• Hamming Codes
• Bose-Chaudhuri-Hocquenhem BCH Codes
• Reed-Solomon Codes
• Convolutional Codes

20
Example (n,k) Hamming Code
(n,k) (2m-1,2m-1-m), for any m
21
(7,4) Hamming Code
Corrects all single errors
22
Parity Check Matrix
23
Decoding
24
(7,4) Messages and Codes(Systematic Code)
0 0 0 0 0 0 0
1 0 0 1 0 0 0 1
1 0 1 0 0 0 1 0
1 0 1 1 0 0 1 1
1 1 0 0 0 1 0 0
1 1 0 1 0 1 0 1
1 1 1 0 0 1 1 0
1 1 1 1 0 1 1 1
1
0 0 0 0 0 0 0 1
0 1 0 0 0 1 1 1
1 0 0 1 0 0 1 0
0 0 1 1 0 1 1 0
1 0 0 1 1 0 0 1
0 1 1 0 0 0 1 1
0 0 0 1 0 1 1 1
1 1 0 1 0 0 0 0
1 1 1 0 0 1 0 0
1 1 0 1 0 1 0 0
1 0 1 1 1 0 1 1
1 0 0 0 0 0 1 1
0 1 0 1 0 1 1 1
0 1 1 1 1 1 1 1
Messages
Code Words
25
Received Code Words
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 1
1 1 0 0 0 1 1 1
0 0 0 1 0 0 1 0
1 0 1 1 0 1 1 0
0 0 0 1 1 0 0 1
1 1 1 0 0 0 1 1
1 1 0 1 0 1 1 1
1 0 0 1 0 0 0 0
1 0 1 0 0 1 0 0
1 0 0 1 0 1 0 0
1 1 1 1 1 0 1 1
1 1 0 0 0 0 1 1
0 0 1 1 0 1 1 1
0 1 0 1 1 1 1 1
0 0 0 0 0 0 0 1
0 1 0 0 0 1 1 1
1 0 0 1 0 0 1 0
0 0 1 1 0 1 1 0
1 0 0 1 1 0 0 1
0 1 1 0 0 0 1 1
0 0 0 1 0 1 1 1
1 1 0 1 0 0 0 0
1 1 1 0 0 1 0 0
1 1 0 1 0 1 0 0
1 0 1 1 1 0 1 1
1 0 0 0 0 0 1 1
0 1 0 1 0 1 1 1
0 1 1 1 1 1 1 1
Received Words
errors
Transmitted Words
26
Syndromes
1 0 0 0 1 0 0 0
1 1 1 0 0 1 1 1
1 1 1 0 1 1 0
0 0 1 0 0 0 1 1
1 0 0 1 1 1 1 1
1 0 1 1 0 0 0 1
0
4 2 1 6 3 7 5
4 2 1 6 3 7 5
4 2
Syndrome Table
0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 1
0 0 0 0 0 0 0 0
0 1 0 0 1 0 0 0
0 0 0 0 0 0 0 0
0 1 0 0 0 1 0 0
0 0 0 0 0 0 1 0

• Rows in Syndrome Table indicate error locations
• Syndromes point to rows

Binary
Decimal
27
Performance

• Bit Error Probability p10-6
• M 100 Bits
• Probability that message is received correctly,
• Pc(1-10-2)100 .367
• Use a hamming (7,4) code
• 25 7-Bit code words with 4 info bits per word
• Probability that a word is correctly decoded
• Pw probability that one or fewer bits are
  decoded incorrectly
• (1-10-2) 7 (10-2) .998
• Probability that message is received correctly
• (.998)25 .95
• A substantial improvement

28
Video Streaming Architecture
29
Scalable and Non-Scalable Coder/Decoder
Scalable Encoder/Decoder
Scalable Encoder/Decoder

• A nonscalable video encoder generates one
  compressed bit-stream.
• Scalable video encoder compresses a raw video
  sequence into multiple sub-streams

30
Video Stream Architecture Building Blocks

• Video Compression
• Application Layer QoS Control
• Continuous Media Distribution Services
• Streaming Servers
• Media Synchronization
• Protocols for Multimedia

31
Video Compression

• Video compression schemes can be classified into
  two categories scalable and nonscalable
• Scalable video is capable of gracefully coping
  with the bandwidth fluctuations in the Internet.
  
• Primarily concerned with scalable video coding
  techniques. We will also discuss the requirements
  imposed by streaming applications on the video
  encoder and decoder.

32
Application Layer QoS Control

• Congestion control takes the form of rate
  control.
• Three kinds of rate control
• 1) Source-based,
• The source-based rate control is suitable for
  unicast
• Most recent studies on source-based rate control
  have been focused on TCP-friendly adaptation
• A number of TCP-friendly adaptation schemes have
  been proposed and demonstrated to achieve certain
  degree of fairness among competing connections,
  including TCP connections.
• However, strictly TCP-like rate control may
  result in sharp reductions in the transmission
  rate, and possibly unpleasant visual quality
  66.
• Needs further investigation on how to trade off
  responsiveness in detecting and reacting to
  congestion with smooth fluctuation in visual
  quality.
• 2) Receiver-based, and 3)Hybrid rate-control.
• Both are suitable for multicast since both can
  achieve good tradeoff between bandwidth
  efficiency and service flexibility for multicast
  video.

33
Source Based Rate Control
34
Continuous Media Distribution Services

• Continuous media distribution services are built
  on top of the best-effort Internet with the aim
  of achieving QoS and efficiency for streaming
  video.
• A major topic of active research is how to build
  a scalable, efficient, cost-effective and
  incremental deployable infrastructure for
  continuous media distribution.

35
Streaming Servers

• Essential in providing streaming services.
• Current research efforts include
• 1) Efficient support VCR-like interactive
  control
• 2) The design of efficient and reliable storage
  and retrieval of multimedia objects on disk
  arrays
• 3) The design of highly scalable multimedia
  servers in a variety of environments ranging from
  video-on-demand servers to integrated multimedia
  file system
• 4) The design of fault-tolerant storage systems
• Desirable features of both parity and mirroring
• Trade off the parity group size with the number
  of disks across which original data of a single
  disk is replicated for mirroring.

36
Media Synchronization

• Intra-stream synchronization The lowest layer of
  continuous media or time-dependent data (such as
  video and audio) is the media layer.
• Maintains the continuity of logical data units.
• Prevents pauses and gaps.
• Inter-stream synchronization The second layer of
  time-dependent data is the stream layer.
• Maintains temporal relationships among different
  continuous media.
• Prevents skew between the streams may become
  intolerable.
• eg, users could be annoyed if they notice that
  the movements of the lips of a speaker do not
  correspond to the presented audio.
• Inter-object synchronization The highest layer
  of a multimedia document is the object layer,
  which integrates streams and time-independent
  data such as text and still images.
• Synchronization at this layer is referred to as
  inter-object synchronization.
• The objective of inter-object synchronization is
  to start and stop the presentation of the
  time-independent data within a tolerable time
  interval,
• e.g., the audience of a slide show could be
  annoyed if the audio is commenting one slide
  while another slide is being presented

37
Protocols for Multimedia

• Network Layer Protocol
• Transport Protocol
• Session Control Protocol
• UDP
• TCP
• RTP
• RTCP

38
Standardized Protocols

• Several protocols have been standardized for
  communication between clients and streaming
  servers.
• Future research topics on design of protocols
  include
• 1) How to take caches into account (e.g., how to
  communicate with continuous media caches and how
  to control continuous media caches)
• 2) How to efficiently support pause/resume
  operations in caches (since the pause/resume
  operations interfere with the sharing of a
  multimedia stream among different viewers) and
• 3) How to provide security in the protocols

39
Unicast and Multicast
Multicast video distribution using
point-to-multipoint transmission.
Unicast video distribution using multiple
point-to-point connections.
40
Streaming Server Components

• Communicator
• A communicator involves the application layer and
  transport protocols implemented on the server.
• Through a communicator, the clients can
  communicate with a server and retrieve multimedia
  contents in a continuous and synchronous manner.
• Operating system
• Different from traditional operating systems,
• An operating system for streaming services needs
  to satisfy real-time requirements for streaming
  applications.
• Storage system
• A storage system for streaming services

41
Protocol Stack for Multimedia
42
RTP

• RTP does not guarantee QoS or reliable delivery,
  but rather, provides the following functions in
  support of media streaming
• Time-stamping RTP provides time-stamping to
  synchronize different media streams.
• Sequence numbering RTP employs sequence
  numbering to place the incoming RTP packets in
  the correct order. Since packets arriving at the
  receiver may be out of sequence (UDP does not
  deliver packets in sequence),.
• Payload type identification The type of the
  payload contained in an RTP packet is indicated
  by an RTP-header field called payload type
  identifier.
• The receiver interprets the content of the packet
  based on the payload type identifier.
• Certain common payload types such as MPEG-audio
  and video have been assigned payload type numbers
  
• For other payloads, this assignment can be done
  with session control protocols.
• Source identification The source of each RTP
  packet is identified by an RTP-header field
  called Synchronization Source identifier (SSRC),
  which provides a means for the receiver to
  distinguish different sources.

43
RTCP

• QoS feedback This is the primary function of
  RTCP.
• RTCP provides feedback to an application
  regarding the quality of data distribution.
• The feedback is in the form of sender reports
  (sent by the source) and receiver re-ports (sent
  by the receiver).
• The reports can contain in-formation on the
  quality of reception such as
• 1) Fraction of the lost RTP packets, since the
  last report
• 2) Cumulative number of lost packets, since the
  beginning of reception
• 3) Packet interarrival jitter and
• 4) Delay since receiving the last senders
  report.
• Participant identification A source can be
  identified by the SSRC field in the RTP header.
• RTCP provides a human-friendly mechanism for
  source identification.
• RTCP SDES (source description) packets contain
  textual information called canonical names as
  globally unique identifiers of the session
  participants.
• It may include a users name, telephone number,
  email address, and other information.

44
RTCP

• Control packets scaling To scale the RTCP
  control packet transmission with the number of
  participants, a control mechanism is designed as
  follows.
• The control mechanism keeps the total control
  packets to 5 of the total session bandwidth.
• Among the control packets, 25 are allocated to
  the sender reports and 75 to the receiver
  reports.
• To prevent control packet starvation, at least
  one control packet is sent within 5 s at the
  sender or receiver.
• Inter-media synchronization RTCP sender reports
  contain an indication of real time and the
  corresponding RTP timestamp. This can be used in
  inter-media synchronization like lip
  synchronization in video.
• Minimal session control information. This
  optional functionality can be used for
  transporting session information such as names of
  the participants.

45
Session Control Protocols

• RTSP functions
• Support VCR-like control operations such as stop,
  pause/resume, fast forward, and fast backward.
• Provides a means for choosing delivery channels
  (e.g., UDP, multicast UDP, or TCP), and delivery
  mechanisms based upon RTP.
• RTSP works for multicast as well as unicast.
• Also establishes control streams of continuous
  audio and video media between the media servers
  and the clients.
• Specifically, RTSP provides the following
  operations.
• Media retrieval The client can request a
  presentation description, and ask the server to
  setup a session to send the requested media data
• Adding media to an existing session The server
  or the client can notify each other about any
  additional media becoming available to the
  established session

46
Session Control Protocols

• Similar to RTSP, SIP can also create and
  terminate sessions with one or more
  participants.
• Unlike RTSP, SIP supports user mobility by
  proxying and redirecting requests to the users
  current location

47
References

• D. Wu et. Al. Streaming Video over the Internet
  Approaches and Directions, IEEE Transactions on
  Circuits and Systems for Video Technology, Vol.
  11, No.3, March 2001