Video Communications over Wireless and Wired Networks

1
Video Communications over Wireless and Wired
Networks
Video and Image Processing Lab UC Berkeley
2
Outline

• Best effort wired networks
• consider compression and transport together
• Unicast and multicast
• Experimental results
• Extensions
• QoS enabled networks
• Wireless networks
• compression techniques
• link/network/transport protocols

3
Problem Overview

• Real-time video delivery over best-effort
  Internet
• Need RATE-CONTROL for dynamic sharing
• Need ERROR-RESILIENCE
• Real-time constraint - no retransmissions
• Bursty traffic - FEC ineffective
• Supports video conferencing, interactive access
  of stored video, etc.
• No changes of current infrastructure
• Exploit interplay between compression/transport

4
Solution Overview

• Rate-Control
• Adaptive Transport Protocol
• Dynamically shares bandwidth
• Imposes transmission budget on video source
• Scalable Compression
• No transcoding - Low latency complexity
• Error-Resilience
• Individually decodable packets - eliminates error
  propagation
• Equally important packets - minimize quality
  variability under loss

5
Solution Overview (contd)

• What is scalable video?
• Embedded representation - subsets corresponding
  to same video at lower rates
• Why scalability?
• Rate-shaping via simple filtering - low
  latency/complexity
• Allows decoding on machines with limited power
• Why not MPEG?
• Pre-compressed MPEG video is unable to respond to
  dynamic transmission rate imposed by rate control
• Severe error propagation from packet loss
• Scalable MPEG typically generates only 2-3 rates

6
Resilience to Random Loss
Probability
Decodable packets
N 20, p 0.1
N linearly dependent packets
7
Resilience to Random Loss (Contd)
N independent packets
N 20, p 0.1
8
Implementation - Overview

• Compression overview
• 3D subband coding
• data partitioning in frequency domain
• multi-rate quantization
• layered block coding
• many very fine layers, e.g., 30 at 100 kbps each
• Unicast Transport overview
• rate-based protocols
• TCP-friendly over larger time-scale for reduced
  variability in data rate

9
Implementation - Overview
Subbands
Data Partitioning
3D Subband Analysis
Partitions
Network
Reconstructed Video
3D Subband Synthesis
Data Joining
10
Achieving Error Resilience

• Partitions coefficients into groups of blocks
• Each group gets a block from every subband but in
  different spatial location
• Groups compressed independently
• One packet formed from each group
• Error resilience comes from the ability to decode
  packets independently
• Packets are equally important
• Disperse effects of packet loss

Partitioning into groups of blocks the gray
blocks are compressed together.
11
Achieving Rate Scalability
LSB
MSB
R3
LSB
coefficient blocks from a partition
MSB
R2
R1
LSB
MSB

• Order bit-planes by energy reduction per bit
• Fine rate-scalability achieved by preferential
  truncation of bit-planes

12
Progressive Quantization - Implementation (ICIP
94)

• Subbands progressively quantized
• Decide if each pixel is quantized to zero
• compress significance map using layered block
  coding

13
Coding Significance map- Implementation (ICIP 96)
8 x 8 sum block
4 x 4 sum block
2 x 2 sum block
Sum of all pixel values
16 x 16 block of binary pixel

• Quad-tree coding for first significant layer
• Short codewords for blocks with many zeros
• Efficient for high frequency subbands

14
Layered Block Coding - Implementation (ICIP 96)
Insignificant pixel
Significant pixel
Significance map of first significant bit-plane
Significance map of next bit-plane
Layer 0 bits 1000 0100 1101
Layer 1 bits -000 0-01 --0- 0101

• Significance map only grows with more bit-planes
• Skip information implicit in previous bit-planes

15
A Famous Example
Lena compressed at 0.3 bpp and subjected to 20
random packet loss
Original Lena
16
Unicast Transport - Overview

• Common transport protocols
• TCP - reliable, not real-time
• UDP - real-time, unreliable, non-adaptive
• Adaptive transport protocols
• adjust sending rate according to changing channel
  condition
• real-time, unreliable, adaptive
• TCP friendly protocol

17
Video using TCP ?

• Achieves reliability by retransmission
• non-realtime variable/high delay
• increasing delay for material coded at higher
  rate than the connection allows
• Achieves resource sharing by rate adaptation
  exponential back-off, linear increase
• drastic changes in instantaneous bandwidth

18
Video using UDP?

• Fastest possible delivery
• Unreliable
• No rate control
• can lead to congestion collapse as number of
  senders increases
• unfairly consumes more bandwidth when other
  protocols reduces transmission rates in face of
  congestion

19
Adaptive Transport

• Desired cooperative protocols
• share bandwidth fairly with itself
• share bandwidth fairly with TCP
• Desired real-time protocols with relatively
  steady transmission rate
• Solution build TCP-friendly rate controller on
  top of UDP

20
TCP Friendliness

• Estimate average bandwidth of a TCP connection
  under observed traffic conditions
• Need to estimate
• round trip time (rtt)
• probability of burst loss (p)
• number of bursts / throughput, over an
  observation window (32 - 128 rtt).
• Larger window gives smoother rates but slower
  response to changes

21
TCP Friendliness

• Average bandwidth

• MTU Maximum Transport Unit, a constant set to
  500 bytes in our experiments
• Floyd et. al. 1997

22
Results Overview

• Compression Performance
• Speed Performance
• Performance under uniform loss (simulated)
• Internet Unicast Experiments (HK-Berkeley,
  Berkeley-Toronto)
• TCP
• Uncontrolled UDP
• Verifying TCP-friendly protocol
• TCP-friendly protocol non-resilient coding
• TCP-friendly protocol non-resilient coding
  adaptive FEC
• TCP-friendly protocol our resilient coding

23
Compression Performance
PSNR

• Comparable to H.263
• Use 320x224 images
• 600 frames fighting scene from Raiders of the
  lost Ark
• 300 frames from Mother and Daughter
• Both schemes use GOP of 4

Bit rate (kbps)
24
Speed Performance

• Real-time in current workstations (170 MHz
  Ultra-Sparc)
• Employing 4 levels of spatial and 2 levels of
  temporal subband decomposition

25
Performance under Loss

• Raiders sequence at 800 kbps subjected to
  simulated 5 random packet loss
• Considers 5 schemes
• proposed scheme with error resilient
  packetization and 250 byte packets (GOP 4)
• subband coding with 1 packet per subband (GOP 4)
• subband coding scheme (ICIP 96) so that packet n
  depends on packet n-1 etc. (GOP 4).
• MPEG-1 with GOP 2 packetized according to RFC2250
  with average packet size 250 bytes. (do not split
  a slice across packets if possible)
• MPEG-1 with GOP 15 similarly packetsized

26
Performance - 5 loss
Proposed Scheme
1 subband per packet
MSE
ICIP-96
MPEG GOP-2
MPEG GOP-15
25
Time (s)
27
Video Using TCP(Trace Toronto to Berkeley,
noon, 5/11/98)
300
Time
- Application delay can be significant even when
average throughput is higher than data rate
under TCP.
28
Video using UDP?(Trace Hongkong to Berkeley, 2
pm, 9/8/98)
300
Time
- Uncontrolled flows can cause severe congestion
for connections with little bandwidth.
(Scheme ICIP 96)
29
Video using UDP?(Trace Toronto to Berkeley,
noon, 5/12/98)
300
- Congestion still occurs for higher bandwidth
connections when bit-rate is high (Scheme ICIP
96) - Conclusion uncontrolled flows are
undesirable
30
TCP-Friendliness

• Throughput for concurrent transmission Toronto to
  Berkeley

31
TCP-Friendliness

• Throughput for concurrent transmission Hong Kong
  to Berkeley

32
TCP Friendliness(Toronto - Berkeley, 2 pm, May
8, 1998)
- Extra traffic started at time 80 seconds
causes transient losses and increase in rtt -
Losses remain low after protocol reduces
transmission rate
33
Video QualityRate control non-resilient
coding (ICIP 96)
300
Time
- Hong-Kong to Berkeley at 3 pm, 10/20/98 -
Bursty losses cause high distortion due to error
propagation - Average distortion 244
34
Video QualityRate control resilient coding
300
Time
- Hong-Kong to Berkeley at 3 pm, 10/20/98 - Lower
average distortion 105 - Less variability in
quality
35
Adaptive FEC

• Given a fixed bit budget, give up less important
  packets in favor of duplicates of more important
  packets
• Given fixed rate-distortion characteristics and
  dynamic estimates of channel, decide the optimal
  set of packets to transmit
• Problem bursty traffic / delayed estimates

36
Video QualityRate control non-resilient
coding FEC
300 s
Time
- Hong-Kong to Berkeley at 3 pm, 10/20/98 -
Adaptive unequal error protection - Fewer
catastrophes - Average distortion 201
37
Video Quality - Review
Resilient Video Flow Control
300 s
300 s
Non-resilient Video Flow Control
300 s
Non-resilient Video FEC Rate Control
38
Video Multicast with Hierarchical FEC

• Problem different IP-multicast users experience
  different channel characteristics and have
  different tolerance for latency.
• Goal Flexible, low cost/complexity framework to
  allow different users to trade-off latency for
  better video quality.

39
Solution Overview

• Hierarchical FEC
• Code FEC data in hierarchical/layered manner
• Carry each layer using a different multicast
  group (address)
• Each receiver individually decides how many
  layers to subscribe based on past history
• Introduce latency to FEC layers to achieve
  interleaving Bolot 1997

40
Hierarchical FEC - Timing Diagram
41
Hierarchical FEC - Example
Video Data
Interactive User
FEC layer 1
FEC layer 2
Passive viewer, no loss
Source
Passive viewer, low loss
Passive viewer, high loss
42
Producing Hierarchical FEC

• Maximal Distance Separable (MDS)
• (punctured) Reed-Solomon
• No loss in error correcting capability in
  imposing the hierarchical structure.

43
Hierarchical FEC - Example

• (77, 70) punctured Reed-Solomon code over
  GF(256).
• The 7 parity packets can be distributed into FEC
  layers, so that users who want protection up to
  10 can subscribe to all layers.
• For 1 Mbps bit-stream, FEC encoding and decoding
  of the above code adds roughly a 5 CPU load on a
  Pentium 400.

44
Resilient Video Multicast
Layer 4
Layer 3
Layer 2
Layer 1
Time ?
Time 2?
Time 0
Useful Packet
Useless Packet
Packet Loss
45
Experimental Setup
Receiver Gatech

• Experiment 1
• No FEC used
• Experiment 2
• Hierarchical FEC used
• high motion video clip
• 320x224, 12 fps
• every 4 frames coded together

6
7
5
Source Berkeley
Receiver ISI
46
Result Observed loss rate

• Berkeley - Gatech
• no observed loss
• Berkeley - ISI
• Similar loss rate across layers and experiments
• Average 18 loss

Observed packet loss rates for different data
layers (Berkeley - ISI)
47
Result Video Traces
MSE
MSE trace without FEC (Berkeley-ISI)
MSE
MSE trace with hierarchical FEC (Berkeley-ISI)
48
Result Loss Trace (Exp 2)
Packet reception rate

• Packet loss measured over 1/3 s intervals
• Bursty varies from 0 to over 90

49
Future Work on Wired Best Effort Networks

• Unicast
• Simulations needed to verify the dynamics of rate
  control window size for estimating loss and
  round trip time, and update frequency
• Multicast
• Simulations needed to verify the dynamics of rate
  control
• stability - coordination between receivers

50
Future QoS and Wireless

• Extend from best effort networks to QoS enabled
  networks.
• Extend the TCP-Friendly transport protocol from
  wired to wireless
• exploit existing work on mobile TCP
• Develop new compression techniques (MDC) for
  wireless systems with diversity, e.g. Multiple
  Element Antenna (MEA)

51
Best Effort versus QoS

• So far, only packet switched networks with best
  effort.
• Future involves Quality of Service type
  networks.
• Examples Internet2, NGI, Calren2,
• RSVP
• Differentiated Services

52
RSVP

• Introduced in the eighties
• Requires users to reserve resources to guarantee
  certain bit rate, delay and loss
• Introduces state in the network.
• Not as robust to node failures as before
• Does NOT scale well with the number of users and
  size of the network.

53
Differentiated Services

• Currently under discussion at IETF
• Differentiation between premium and
  non-premium services.
• Define premium in terms of delay/loss/jitter
• Use the type of service field in IP to carry
  information about service requirements
• Routers check TOS to process each packet

54
How to exploit DiffServ for video communication

• Decompose video into multiple streams with
  different QoS requirements
• Example low/high delay low/high loss low/high
  bandwidth
• Mark the packets corresponding to different
  streams differently at the edge so they are
  processed accordingly

55
Design Issues

• How to optimally decompose video into multiple
  streams without loss of compression efficiency?
• Delay Cognizant Video (DCV) Yuanchi and
  Messerschmitt 1998.
• Error resilient, rate scaleable video Tan and
  Zakhor 1998

56
Design Issues

• Assured Services Clark
• traffic unlikely to be dropped if within expected
  capacity profile. (unlikely needs to be defined)
• Premium Services Jacobson
• traffic shaped to its provisioned peak rate
• Issue Optimal video decomposition depends on the
  exact form of diffserv

57
Wireless Multimedia Communications

• MM requirements different from data
• delay sensitive
• massively compressed
• could be sensitive to loss
• could also have graceful degradation
• unlike data BER is not an indicative of
  performance audio/visual quality is.
• bits of unequal importance

58
Existing Approaches

• Shannons separation theorem does not hold
• states that source and channel coding can be done
  separately without loss in performance.
• only a mathematical proof doesnt actually tell
  you how to do it.
• does not take into account delay or complexity

59
Existing Approaches

• Joint source channel coding (JSCC)
• Digital JSCC
• allocates bits optimally between source and
  channel coding
• unequal error protection
• Analog JSCC
• maps source codewords onto channel constellation
  points

60
JSCC

• Channel models
• Discrete memoryless BSC
• Additive White Gaussian Noise (AWGN)
• Directly model physics of system
• average/peak transmission power
• delay
• bandwidth

61
Diversity

• Commonly used technique in wireless
  communications
• Comes in three flavors
• frequency/spectral diversity
• temporal diversity
• spatial diversity
• Multiple Element Antenna (MEA)
• either at transmitter or at receiver

62
Proposed Idea

• Apply JSCC to multimedia wireless communication
  systems that exploit diversity.
• Question What sort of JSCC?
• Multiple Description Coding (MDC) of multimedia
  content
• Question What sort of diversity?
• Spatial diversity with MEA arrays.

63
Multiple Description Coding

• Invented in the eighties.
• Basic idea of MDC
• Decompose the source into multiple bit streams
• Send each bit stream across a different channel
• Different channels are statistically independent
  
• More streams at the receiver, the better quality
• Not all channels bad simultaneously
• Graceful degradation of quality video/audio

64
Example of MDC

• Video source.
• Decompose it into two streams
• Send streams across two different routes in a
  packet switched network, e.g. two ISPs
• If one stream faces congestion, the other will
  result in reasonable quality
• If both streams make it, excellent quality

65
An Example of MDC Quantizer

• Overlap the quantization bins of two quantizers.

Q1
Q2
Combined rate of two quantizers is larger
than single description coding with one quantizer
66
MDC

• Fundamental problem
• Given a source with rate distortion
  characteristics (R,D), what are the achievable
  rate distortion characteristics of the
  substreams, e.g. (R1,D1) and (R2,D2)?
• what is the price to be paid for having multiple
  stream in terms of additional rate, or additional
  distortion?

67
MDC

• Plenty of approaches proposed for images
• Reibman, Orchard, Wang 1997, 1998
• Basic idea
• orthogonal transformation
• MDC quantization of the transform coefficients
• introduce deliberate correlation among transform
  coefficients of different blocks
• No existing MDC technique for video yet

68
MDC and JSCC

• Can MDC be used as a form of JSCC?
• Deliberate introduction of correlation between
  the two streams is a way of protecting them
  against loss gt Channel coding.
• Can even control the level of redundancy between
  the two streams accurately!

69
MDC and Diversity

• Can MDC be used with diversity?
• Natural fit with MEA
• Can send each streams of MDC source onto a
  different channels of the MEA
• In MDC transmitter does NOT have to know the
  conditions of the channel
• No feedback required

70
Design Issues

• Given statistical characterizations of the
  available MEA channels and their correlation,
  what is the optimum level of correlation that
  should be introduced among the multiple
  descriptions?

71
Design Issues

• How should one exploit the intentional
  correlation between the multiple streams
  introduced at the source coding stage, and the
  unintentional correlation among different
  channels in order to arrive at optimal
  detection/estimation algorithms?

72
Design Issues

• Can space time codes for MEA be optimally
  combined with MDC schemes?
• What are the criteria?
• What are some efficient MDC coding schemes for
  video?
• Overcomplete dictionaries, MP?
• What about motion prediction/compensation

73
Design Issues

• What happens if the transmitter transmits at
  rates higher than capacity of the actual system
  to do mis-estimation of the correlation between
  channels?
• Exploit correlation among streams to derive
  detection/estimation algorithms that detect a
  subset of the information?

74
Summary

• Addressed wired, best effort case.
• Issues to be addressed
• extensions to QoS enabled networks
• extensions of link/transport real time multimedia
  protocols to wireless.
• Develop MDC video compression schemes to be
  combined with spatial diversity wireless systems
  such as MEA.