Duality between Source Coding and Channel Coding

1
Duality between Source Coding and Channel Coding

• Nan Ma and Wei Kang
• Directed by Prof. Ishwar
• 4/26/2006

2
Outline

• Motivation
• Duality between conventional source coding and
  channel coding
• Duality and optimality
• Extension to source coding and channel coding
  with side information
• Conclusion

3
Motivation

• Similarity between source coding (SC) and channel
  coding (CC)
• SC to minimize the mutual information under
  distortion constraint
• CC to maximize the mutual information under cost
  constraint

4
Motivation (2)

• Shannons word (1959)
• This duality can be pursued further and is
  related to a duality between past and future and
  the notions of control and knowledge. Thus, we
  may have knowledge of the past but cannot control
  it we may control the future but not have
  knowledge of it.

5
Outline

• Motivation
• Duality between conventional source coding and
  channel coding
• Duality and optimality
• Extension to source coding and channel coding
  with side information
• Conclusion

6
Symbols

• Given distribution
• Optimal solution
• Input of SC, output of CC
• Input of CC, output of SC
• Cost constraint w, W
• Distortion measure d, D

7
Fact 1 Given quantizer, can find distortion
Quantizer
Distortion

• Recall calculating rate-distortion function
• Fact 1 says

s.t.
8
From fact 1 to Duality theorem 1 (CC-gtSC)
Cost
Channel
Quantizer
Distortion
9
Theorem 1 (CC-gtSC) (continued)

• Distortion measure is determined by the dual
  channel
• Rate-distortion vs. capacity-cost

This is the dual channel!
10
Fact 2 Given channel input, can find cost
measure
Channel
Cost

• Recall calculating channel capacity
• Fact 2 says

s.t.
11
From fact 2 to Duality theorem 2 (SC-gtCC)
Cost
Channel
Quantizer
Distortion
12
Theorem 2 (SC-gtCC) (continued)

• Cost function is determined by the dual source
• Capacity-cost vs. rate-distortion

This is the dual source!
13
When SC and CC are duals
Cost
CC
SC
Distortion
Given , can we find SC and CC
duals?
14
Backward channel of SC
Cost
Channel
Identical except constraint
Quantizer
Distortion
Recall the example of test channel!
15
Recall mutual information game (MIG)

• X and Z are independent
• Constraint on signal and noise power
• EX2 P, EZ2 N
• Noise player tries to minimize I(X Y)
• Signal player tries to maximize I(X Y)

16
From SC viewpoint

• We fix p(X) noise player Z ? a quantizer
• Power constraint on Z ? the quadratic distortion
  measure.
• The optimal quantizer is Guassian

17
From CC viewpoint

• We fix p(Z) noise player Z? a channel
• Fixing p(Z) ? p(ZX) ? p(YX)
• Power constraint on X ? the quadratic cost
  constraint.
• The optimal input is Guassian

18
Is MIG a dual of itself?

• XN(0, P), ZN(0, N) ? R(N) C(P)
• Joint distribution are the same!
• However, input of SC should be output of CC dual
  and vice versa
• MIG might suggest that the dual of Gaussian SC is
  a Gaussian channel.

19
Example of duality (1)

• CC
• Gaussian channel
• cost
• optimal input

• SC
• Gaussian source
• distortion
• optimal quantizer

20
Example of duality (2)

• Bluhat-Arimoto Algorithm (Numerical)
• Given SC
• problem
• Get Dual
• CC problem

21
Shannons oracle

• we may have knowledge of the past but cannot
  control it
• Source coding problem

Quantizer
Distortion
22
Shannons oracle (continued)

• we may control the future but not have
  knowledge of it.
• Channel coding problem

Channel
Cost
23
Outline

• Motivation
• Duality between conventional source coding and
  channel coding
• Duality and optimality
• Extension to source coding and channel coding
  with side information
• Conclusion

24
Optimality
S
X
Y
Source
Channel
F
G
Destination

• General idea, source-channel code (F, G) is
  optimal iff
• Fixed input cost W , distortion D is minimized,
• Fixed distortion D, input cost W is minimized.

25
Single-letter code
S
X
Y
Source
Channel
F
G
Destination

• Single-letter code is optimal if
• R(D) C(W)
• or equivalently,

Duality!
26
Single-letter code

• Duality
• Capacity Rate
• The channel can transfer exactly what the source
  wants optimality!

Identical
X
Y
S
Channel
Backward Channel
27
Outline

• Motivation
• Duality between conventional source coding and
  channel coding
• Duality and optimality
• Extension to source coding and channel coding
  with side information
• Conclusion

28
Source Coding with Side Information
U
X
Encoder
S

• Given a source p(ux), side information p(s),
  distortion measure d, we can have

29
Source Coding with Side Information

• Then we can calculate the joint distribution
• And more

30
Source Coding with Side Information

• If
• Then we can find a channel coding with side
  information

31
Source Coding with Side Information

• with certain conditional distributions and cost
  constraint

32
Outline

• Motivation
• Duality between conventional source coding and
  channel coding
• Duality and optimality
• Extension to source coding and channel coding
  with side information
• Conclusion

33
Conclusion

• Explore the basic concept of duality for
  conventional SC and CC
• Describe several simple examples, such as
  Gaussian SC vs. Gaussian channel, MIG, BA
  Algorithm
• Describe optimality for single-letter code and
  the connection with duality

34
Thank you!