Towards Practical Distributed Coding

1
Towards Practical Distributed Coding
Bernd Girod Information Systems
Laboratory Stanford University
2
Outline

• Distributed lossless compression
• Simple examples
• Slepian-Wolf Theorem
• Slepian-Wolf coding vs. systematic channel coding
• Turbo codes for compression
• Lossy compression with side information
• Wyner-Ziv Theorem
• Optimal quantizer design with Lloyd algorithm
• Selected applications
• Sensor networks
• Low complexity video coding
• Error-resilient video transmission

3
Simple Example
Source A
4
Simple Example - Revisited
Source A
5
Compression with Side Information
Source A/B
Encoder
Decoder
6
Distributed Compression of Dependent Sources
Source X
Encoder X
X
Joint Decoder
Y
Source Y
Encoder Y
7
Achievable Rates for Distributed Coding Example
8
General Dependent i.i.d. Sequences
Slepian, Wolf, 1973
9
Distributed Compression and Channel Coding
Source XY
Encoder
Decoder
P

• Idea
• Interpret Y as a noisy version of X with
  channel errors D
• Encoder generates parity bits P to protect
  against errors D
• Decoder concatenates Y and P and performs
  error-correcting decoding

10
Practical Slepian-Wolf Encoding

• Coset codes Pradhan and Ramchandran, 1999
• Trellis codes Wang and Orchard, 2001
• Turbo codes
• Garcia-Frias and Zhao, 2001
• Bajcsy and Mitran, 2001
• Aaron and Girod, 2002
• LDPC codes Liveris, Xiong, and Georghiades,
  2002

11
Compression by Turbo Coding
L bits in
L bits
Systematic Convolutional Encoder Rate
Systematic Convolutional EncoderRate
L bits
Aaron, Girod, DCC 2002
12
Turbo Decoder
Pchannel
SISO Decoder
Pa posteriori
Pextrinsic
Pa priori
L bits out
Pextrinsic
Pa priori
SISO Decoder
Pchannel
Pa posteriori
Aaron, Girod, DCC 2002
13
Results for Compression of Binary Sequences

• X,Y dependent binary sequences with symmetric
  cross-over probabilities
• Rate 4/5 constituent convolutional codes
• RX0.5 bit per input bit

Aaron, Girod, DCC 2002
14
Results with Puncturing
Aaron, Girod, DCC 2002
15
Channel Coding vs. Slepian-Wolf Coding

• Systematic Channel Coding
• Systematic bits and parity bits subject to
  bit-errors
• Mostly memoryless BSC channel
• Low bit-error rate of channel
• Rate savings relative to systematic bits

• Slepian-Wolf Coding
• No bit errors in parity bits
• General statistics incl. memory
• Whole range of error probabilities
• Rate savings relative to parity bits
• Might have to compete with conventional
  compression (e.g., arithmetic coding)

16
Distributed Lossy Compression of Dependent Sources
Source X
Encoder X
X
Joint Decoder
Y
Source Y
Encoder Y
17
Lossy Compression with Side Information
Source
Encoder
Decoder
18
Practical Wyner-Ziv Encoder and Decoder
Wyner-Ziv Decoder
Wyner-Ziv Encoder
Slepian- Wolf Decoder
Minimum Distortion Reconstruction
Slepian-Wolf Encoder
Quantizer
19
Non-Connected Quantization Regions

• Example Non-connected intervals for scalar
  quantization
• Decoder Minimum mean-squared error
  reconstruction with side information

x
x
20
Quantizer Reconstruction Function
21
Finding Quantization Regions
22
Lloyd Algorithm for Wyner-Ziv Quantizers
Choose initial quantizers
Find best reconstruction functions for current
quantizers

• Fleming, Zhao, Effros, unpublished
• Rebollo-Monedero, Girod, DCC 2003

Lagrangian cost for current quantizers,
reconstructor and rate measure
N
Find best quantizers for current reconstruction
and rate measure
23
Which Rate Measure?
Wyner-Ziv Decoder
Wyner-Ziv Encoder
Slepian- Wolf Decoder
Slepian-Wolf Encoder
Minimum Distortion Reconstruction
Quantizer
24
Example
Data set Video sequence Carphone, 100 luminance
frames, QCIF

• X pixel values in even frames
• Y motion-compensated interpolation from two
  adjacent odd frames

25
Example (cont.)
Quantizer w/ rate constraint H(Q)
Quantizer w/ rate constraint H(QY)
PSNR39 dBH(Q)3.05 bit H(QY)0.54
bit
PSNR37.4 dBH(Q)1.87 bit H(QY)0.54 bit
26
Example (cont.)
Quantizer w/ rate constraint H(Q) Quantizer w/
rate constraint H(QY)
PSNRdB
PSNRdB
Rate bit
Rate bit
27
Wyner-Ziv Quantizers Lessons Learnt

• Typically no quantizer index reuse for rate
  constraint H(QY) and high rates Slepian-Wolf
  code provides more efficient many-to-one mapping
  in very high dimensional space.
• Uniform quantizers close to minimum m.s.e., when
  combined with efficient Slepian-Wolf code
• Quantizer index reuse required for rate
  constraint H(Q) and for fixed-length coding
• Important to decouple dimension of quantizer
  (i.e. scalar) and Slepian-Wolf code (very large)

28
Sensor Networks
Local Sensor
Central Unit
Side Information
Pradhan, Ramchandran, DCC 2000 Kusuma,
Doherty, Ramchandran, ICIP 2001 Pradhan,
Kusuma, Ramchandran, SP Mag., 2002 Chou,
Perovic, Ramchandran, Asilomar 2002
29
Video Compression with Simple Encoder
Aaron, Zhang, Girod, Asilomar 2002 Aaron,
Rane, Zhang, Girod, DCC 2003
30
Video Compression with Simple Encoder
After Wyner-Ziv Decoding
Decoder Side information
16-level quantization (1 bpp)
31
Video Compression with Simple Encoder
After Wyner-Ziv Decoding
Decoder Side information
16-level quantization (1 bpp)
32
Video Compression with Simple Encoder
After Wyner-Ziv Decoding
Decoder Side information
16-level quantization (1 bpp)
33
Performance of Simple Wyner-Ziv Video Coder
34
Digitally Enhanced Analog Transmission
Analog Channel

• Forward error protection of the signal waveform
• Information-theoretic bounds Shamai, Verdu,
  Zamir,1998
• Systematic lossy source-channel coding

35
Forward Error Protection for MPEG Video
Broadcasting
MPEG Encoder
MPEG Decoder with Error Concealment
S
S
Error-Prone channel

• Graceful degradation without a layered signal
  representation

36
Error-resilient Video Transmissionwith Embedded
Wyner-Ziv Codec
Carphone CIF, 50 frames _at_ 30fps, 1 Mbps, 1
Random Macroblock loss
Aaron, Rane, Rebollo-Monedero, Girod, ICIP 2003
37
Towards Practical Distributed CodingWhy Should
We Care?

• Chance to reinvent compression from scratch
• Entropy coding
• Quantization
• Signal transforms
• Adaptive coding
• Rate control
• . . .
• Enables new compression applications
• Sensor networks
• Very low complexity encoders
• Error-resilient transmission of signal waveforms
• Digitally enhanced analog transmission
• Unequal error protection without layered coding
• . . .

38
The End