Error Control and Concealment for Video Communication

1
Error Control and Concealment for Video
Communication
CMPT820 Summer 2008 Michael Jia
2
Reference
Error Control and Concealment for Video
Communication A Review
YAO WANG, Member, IEEEQIN-FAN ZHU, Member, IEEE
3
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

4
Introduction

• 2 Types of Transmission Errors
• Random Bit Errors
• Bit inversion, bit insertion, bit deletion
• Erasure Errors
• Packet loss, burst errors, system failures
• More destructive
• Common usage of VLC makes them no differences in
  video streaming

5
Introduction

• Lossless recovery
• FEC (Forward Error Correction)
• ECC (Error Control Coding)
• ARQ (Automatic Retransmission Request)
• Not necessary for video transmission, human eyes
  can tolerate a certain degree of distortion
• Focus on signal-reconstruction and
  error-concealment techniques

6
Introduction

• Typical structure of a video communication system

7
Introduction

• 3 groups of error-concealment techniques
• Forward error concealment
• Encoder plays the primary role
• Error concealment by post-processing
• Decoder fulfills the task
• Interactive error concealment
• Main concerns
• Effectiveness
• Required delay
• Bit-rate overhead
• Processing complexity

8
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

9
Error Detection

• At transport layer
• Adding header information (sequence number)
• H.223
• FEC (Forward Error Correction)
• H.223, H.261
• More reliable
• Need more bandwidth

10
Error Detection

• At decoder
• Based on characteristics of natural video signals
• Pixel value differences of neighboring lines
  (compare to a threshold)
• Differences between boundary pixels in a block
  and its four neighbor blocks
• Obvious false value of quantization step size or
  DCT coefficients
• Insert synchronization code word at the end of
  line of blocks
• No additional bits or very few
• Rely on smoothness of signal

11
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

12
Error Concealment at Coder (1)

• Layered Coding with Transport Prioritization
• Most popular and effective (MPEG-2)

13
Error Concealment at Coder (1)

• Layered Coding
• Base layer most important layer, with
  acceptable quality
• Transport prioritization
• Deliver base layer with higher degree of error
  protection
• High priority channel
• Re-transmission and/or FEC
• No explicit bit-rate overhead
• Complicate structure and coding overhead
• H.264 AVC/SVC
• Redundant pictures
• Data Partitioning

14
Error Concealment at Coder (2)

• Multiple-Description Coding (MDC)
• Several parallel channels
• Independent error events
• Small probability of all channels down
• Multiple Descriptions
• Several coded bit streams
• Transmitted over separate channels
• Any one will work

15
Error Concealment at Coder (2)

• Multiple-Description Coding (MDC)

16
Error Concealment at Coder (3)

• Joint Source and Channel Coding
• Source-channel interaction at a lower level
• Given channel error characteristics, design
  quantizer and entropy coder for to minimize the
  effect of errors
• For general sources, noisy channel ? coarse
  quantizer is better
• For image signals, noisy channel ?
• fewer bits to high-frequency coefficients
• more bits to low-frequency coefficients

17
Error Concealment at Coder (4)

• Robust Waveform Coding
• Intentionally keep some redundancy in
  source-coding
• Layered coding and MDC both belong to this
  category
• Adding auxiliary information in waveform coder
• MPEG-2 sending motion vectors for microblocks in
  I-frames
• Restricting prediction domain
• H.263/H.264 prediction is confined within each
  slice

18
Error Concealment at Coder (5)

• Robust Entropy Coding
• Add redundancy in entropy-coding
• To help detect bit errors and prevent error
  propagation
• Self-Synchronizing entropy coding
• Add a synchronization code word
• H.261, H.263, MPEG-4
• Error-Resilient entropy coding (EREC)
• MPEG-4 uses RVLC (reversible VLC)

19
Error Concealment at Coder (6)

• FEC Coding
• Guarantee lossless data delivery
• Induct a lot of overhead, reduce usable bandwidth
• Maybe too strong in video services
• Cases
• H.261 in ISDN
• MPEG-2 in wireless ATM local network

20
Error Concealment at Coder (7)

• Transport-Level Control
• Add redundancies at transport level
• Prioritized transport for layered coding
• Robust packetization
• Spatial block interleaving
• Dual transmission of important information
• H.264 AVC/SVC
• NAL unit syntax structure
• Parameter Sets

21
Error Concealment at Coder (Summary)
22
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

23
Error Concealment at Decoder

• Preview
• Perform error concealment at the decoder
• Can be used in conjunction with the auxiliary
  information provided by the source coder
• Low frequency components dominate images of
  natural scenes
• Color values of adjacent pixels vary smoothly
  except sharp edges
• Human eyes tolerate more distortion to
  high-frequency components

24
Error Concealment at Decoder (1)

• Motion-Compensated Temporal Prediction
• Replace damaged MB with the motion compensated
  block
• Very effective when all the motion information in
  the base layer
• Widely used (MPEG-2)
• What if motion information or coding mode is
  lost?
• Will discuss in next slide

25
Error Concealment at Decoder (2)

• Recovery of Motion Vectors and Coding Modes
• What if we lost motion vectors or coding modes?
• Interpolated from spatially and temporally
  adjacent blocks
• Estimation of coding modes
• Simple treat as intracoded
• More sophisticate MPEG-2 (See tables)

26
Error Concealment at Decoder (2)

• Estimation of motion vectors
• Set to zeros (works well for low motion video)
• Use MV of the corresponding block in the previous
  frame
• Use the average of MVs from spatially adjacent
  blocks
• Use the median of MVs from spatially adjacent
  blocks
• Select one of the above methods depending on
  least boundary matching error

27
Error Concealment at Decoder (3)

• Maximally Smooth Recovery
• A constrained energy minimization approach
• Minimize a measure of spatial and temporal
  variation between adjacent pixels in this block
  and its spatially and temporally neighboring
  blocks
• Measure differences in 3 domains
• Spatial adjacent blocks
• Temporal prediction block in previous frame
• Frequency received coefficients for this block

28
Error Concealment at Decoder (3)

• Maximally Smooth Recovery
• 2 sample spatial smoothness measures

29
Error Concealment at Decoder (4)

• Spatial- and Frequency-Domain Interpolation
• A coefficient in a damaged block is likely to be
  close to the corresponding coefficients in
  spatially adjacent blocks
• Interpolate from four neighbor blocks
• Maybe not accurate (8-pixel is too far)
• Interpolate from four 1-pixel-wide boundaries
• 2 pixels in 2 nearest boundaries
• 4 pixels in all 4 boundaries
• See picture in next slide

30
Error Concealment at Decoder (4)

• Spatial- and Frequency-Domain Interpolation
• Interpolate from four 1-pixel-wide boundaries

31
Error Concealment at Decoder (5)

• Projection onto Convex Sets (POCS)
• The convex sets are derived by requiring the
  recovered block to have a limited bandwidth
  either isotropically (for a block in a smooth
  region) or along a particular direction (for a
  block containing a straight edge)
• A combined block is formed by including eight
  neighboring blocks with the damaged block

32
Error Concealment at Decoder (Summary)
33
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

34
Interactive Error Concealment

• Preview
• If a backward channel is available, can achieve
  better performance by cooperation
• Based on the feedback
• At source coder coding parameters can be
  adapted
• At transport level adjust the portion of
  bandwidth used for error control
• Decoding delay issue

35
Interactive Error Concealment (1)

• Selective Encoding for Error Concealment
• Simple code next frame in intramode
• Error stopped in about one round-trip time
• Will cause bit-rate increase
• Send identity info back, perform error
  concealment at the same time
• Continue encode without using the affected area
• Perform same error concealment procedure (need a
  prediction frame buffer)
• See picture in next slide

36
Interactive Error Concealment (1)

• Selective Encoding for Error Concealment

37
Interactive Error Concealment (1)

• Selective Encoding for Error Concealment

38
Interactive Error Concealment (1)

• H.263 uses more prediction frame buffers
  (reference picture selection mode)

39
Interactive Error Concealment (2)

• Adaptive Transport for Error Concealment
• Retransmission is unacceptable for real-time
  video applications?
• Not always
• For one-way video broadcast, we may tolerate a
  few seconds delay
• For multipoint video conferencing, use MCU
  (multipoint control unit)
• If retransmission is controlled properly,
  end-to-end quality can be improved
• Both H.323 and H.324 defined such mechanisms

40
Interactive Error Concealment (3)

• Retransmission Without Waiting
• Wait for the retransmission data
• Not good, may freeze the display
• Cause transit delay and accumulation delay
• Without waiting
• Request the retransmission
• Conceal the error
• Track the affected pixels
• Correct them upon the arrival of the
  retransmission data

41
Interactive Error Concealment (4)

• Prioritized, Multi-copy Retransmission
• Effective in very lossy channels
• Video streaming via PSTN
• Are you kidding me?
• Send multiple copies of a lost packet
• Use layered coding
• of retransmission trials and of copies are
  proportional to the importance of the layer

42
Interactive Error Concealment (Summary)
43
Outline

• Introduction
• Error Detection
• Error Concealment at Coder
• Error Concealment at Decoder
• Interactive Error Concealment
• Conclusion

44
Conclusion

• Real-time video communication doesnt require
  lossless delivery signal-reconstruction and
  error-concealment techniques are more effective.
• Add redundancy when encoding or delivering
• Estimate missing information when decoding
• Inform sender what is lost
• Burstiness has a significant impact on the choice
  of algorithms

45

• Questions?

46

• Thank You