The Role of Specialization in LDPC Codes

1
The Role of Specialization in LDPC Codes

• Jeremy Thorpe
• Pizza Meeting Talk
• 2/12/03

2
Talk Overview

• LDPC Codes
• Message Passing Decoding
• Analysis of Message Passing Decoding (Density
  Evolution)
• Approximations to Density Evolution
• Design of LDPC Codes using D.E.

3
The Channel Coding Strategy

• Encoder chooses the mth codeword in codebook C
  and transmits it across the channel
• Decoder observes the channel output y and
  generates m based on the knowledge of the
  codebook C and the channel statistics.

Encoder
Channel
Decoder
4
Linear Codes

• A linear code C (over a finite field) can be
  defined in terms of either a generator matrix or
  parity-check matrix.
• Generator matrix G (kn)
• Parity-check matrix H (n-kn)

5
LDPC Codes

• LDPC Codes -- linear codes defined in terms of H.
• H has a small average number of non-zero elements
  per row

6
Graph Representation of LDPC Codes

• H is represented by a bipartite graph.
• There is an edge from v to c if and only if
• A codeword is an assignment of v's s.t.

Variable nodes
. . .
. . .
Check nodes
7
Regular (?,?) LDPC codes

• Every variable node has degree ?, every check
  node has degree ?.
• Ensemble is defined by matching left edge "stubs"
  with right edge "stubs via a random permutation

Variable nodes
p
. . .
. . .
Check nodes
8
Message-Passing Decoding of LDPC Codes

• Message Passing (or Belief Propagation) decoding
  is a low-complexity algorithm which approximately
  answers the question what is the most likely x
  given y?
• MP recursively defines messages mv,c(i) and
  mc,v(i) from each node variable node v to each
  adjacent check node c, for iteration i0,1,...

9
Two Types of Messages...

• Liklihood Ratio
• For y1,...yn independent conditionally on x

• Probability Difference
• For x1,...xn independent

10
...Related by the Biliniear Transform

• Definition
• Properties

11
Message Domains
Likelihood Ratio
Probability Difference
Log Likelihood Ratio
Log Prob. Difference
12
Variable to Check Messages

• On any iteration i, the message from v to c is
• It is computed like

v
c
. . .
. . .
13
Check to Variable Messages

• On any iteration, the message from c to v is
• It is computed like
• Assumption Incoming messages are indep.

v
c
. . .
. . .
14
Decision Rule

• After sufficiently many iterations, return the
  likelihood ratio

15
Theorem about MP Algorithm

• If the algorithm stops after r iterations, then
  the algorithm returns the maximum a posteriori
  probability estimate of xv given y within radius
  r of v.
• However, the variables within a radius r of v
  must be dependent only by the equations within
  radius r of v,

r
...
v
...
...
16
Analysis of Message Passing Decoding (Density
Evolution)

• in Density Evolution we keep track of message
  densities, rather than the densities themselves.
• At each iteration, we average over all of the
  edges which are connected by a permutation.
• We assume that the all-zeros codeword was
  transmitted (which requires that the channel be
  symmetric).

17
D.E. Update Rule

• The update rule for Density Evolution is defined
  in the additive domain of each type of node.
• Whereas in B.P, we add (log) messages
• In D.E, we convolve message densities

18
Familiar Example

• If one die has density function given by
• The density function for the sum of two dice is
  given by the convolution

1
3
6
5
4
2
2
4
7
6
5
3
8
10
12
11
9
19
D.E. Threshold

• Fixing the channel message densities, the message
  densities will either "converge" to minus
  infinity, or they won't.
• For the gaussian channel, the smallest (infimum)
  SNR for which the densities converge is called
  the density evolution threshold.

20
Regular (?,?) LDPC codes

• Every variable node has degree ?, every check
  node has degree ?.
• Best rate 1/2 code is (3,6), with threshold 1.09
  dB.
• This code had been invented by 1962 by Robert
  Gallager.

21
D.E. Simulation of (3,6) codes

• Set SNR to 1.12 dB (.03 above threshold)
• Watch fraction of "erroneous messages" from check
  to variable.
• (note that this fraction does not characterize
  the distribution fully)

22
Improvement vs. current error fraction for
Regular (3,6)

• Improvement per iteration is plotted against
  current error fraction.
• Note there is a single bottleneck which took most
  of the decoding iterations.

23
Irregular (?, ?) LDPC codes

• a fraction ?i of variable nodes have degree i. ?i
  of check nodes have degree i.
• Edges are connected by a single random
  permutation.
• Nodes have become specialized.

Variable nodes
p
?2
?4
?3
. . .
. . .
?m
?n
Check nodes
24
D.E. Simulation of Irregular Codes (Maximum
degree 10)

• Set SNR to 0.42 dB (.03 above threshold)
• Watch fraction of erroneous check to variable
  messages.
• This Code was designed by Richardson et. al.

25
Comparison of Regular and Irregular codes

• Notice that the Irregular graph is much flatter.
• Note Capacity achieving LDPC codes for the
  erasure channel were designed by making this line
  exactly flat.

26
Multi-edge-type construction

• Edges of a particular "color" are connected
  through a permutation.
• Edges become specialized. Each edge type has a
  different message distribution each iteration.

27
D.E. of MET codes.

• For Multi-edge-type codes, Density evolution
  tracks the density of each type of message
  separately.
• Comparison was made to real decoding, next slide
  (courtesy of Ken Andrews).

28
MET D.E. vs. decoder simulation
29
Regular, Irregular, MET comparison

• Multi-edge-type LDPC codes improve gradually
  through most of the decoding.
• MET codes have a threshold below the more
  "complex" irregular codes.

30
Design of Codes using D.E.

• Density evolution provides a moderately fast way
  to evaluate single- and multi- edge type degree
  distributions (hypothesis-testing).
• Much faster approximations to Density Evolution
  can easily be put into an outer loop which
  performs function minimization.
• Semi-Analytic techniques exist as well.

31
Review

• Iterative Message Passing can be used to decode
  LDPC codes.
• Density Evolution can be used to predict the
  performance of MP for infinitely large codes.
• More sophisticated classes of codes can be
  designed to close the gap to the Shannon limit.

32
Beyond MET Codes (future work)

• Traditional LDPC codes are designed in two
  stages design of the degree distribution and
  design of the specific graph.
• Using very fast and accurate approximations to
  density evolution, we can evaluate the effect of
  placing or removing a single edge in the graph.
• Using an evolutionary algorithm like Simulated
  Annealing, we can optimize the graph directly,
  changing the degree profile as needed.