LT Codes

1
LT Codes

• Paper by Michael Luby
• FOCS 02
• Presented by Ashish Sabharwal
• Feb 26, 2003 CSE
  590vg

2
Binary Erasure Channel
BEC
decode
encode
Input 00101
Received 10?001??
Input 00101
Codeword 10100101
Packet loss

• Code distance d ) can decode d-1 erasures
• Probabilistic Model
• Bits get erased with prob p
• (Shannon) Capacity of BEC 1 pIn particular,
  pgt1/2 is decodable!

3
LT Codes Encoding
1 code bit
input
1
1 XOR 0 1
1
degree d 2
1
0
1

• Choose degree d from a distribution
• Pick d neighbors uniformly at random
• Compute XOR

4
LT Codes Encoding
codeword
input
1
0
1
1
1
1
1
1
0
1
1
1


5
LT Codes Decoding
codeword
input
?
0
1
1
1
1
1
1
0
1
1
?

• Identify code bit of remaining degree 1
• Recover corresponding input bit

6
LT Codes Decoding
codeword
input
1 0 XOR 1
1
1
1
1
1
1
0
1
1

• Update neighbors of this input bit
• Delete edges
• Repeat

7
LT Codes Decoding
codeword
input
1
1
1
1
1
1
1
0
1
1
8
LT Codes Decoding
codeword
input
0
1
0
1
1
1
0 1 XOR 1
0
1
1
Decoding unsuccessful!
9
LT Codes Features

• Binary, efficient
• Bits can arrive in any order
• Probabilistic model
• No preset rate
• Generate as many or as few code bitsas required
  by the channel
• Almost optimal

RS inefficient
Tornado codes are optimal and linear time, but
have fixed rate
10
Larger Encoding Alphabet

• Why? Less overhead
• Partition input into m-bit chunks
• Encoding symbol is bit-wise XOR

Well think of these as binary codes
11
Caveat Transmitting the Graph

• Send degree list of neighbors
• Associate a key with each code bit
• Encoder and decoder apply the samefunction to
  the key to compute neighbors
• Share random seed for pseudo-randomgenerator

12
Outline

• The Goal
• All 1s distribution Balls and Bins case
• LT Process Probabilistic machinery
• Ideal Soliton Distribution
• Robust Soliton Distribution

13
The Goal

• Construct a degree distribution s.t.
• Few encoding bits required for recovery small
  t
• Few bit operations needed small sum of
  degrees small s

14
All 1s distribution Balls and Bins
All encoding degrees are 1
t unerased code bits
k bit input

• t balls thrown into k bins
• Pr cant recover input
• ? Pr no green input bits
• k . (1 1/k)t
• ¼ k e-t/k
• Pr failure ? d guaranteed if t ? k ln
  k/d

k bins
t balls
15
All 1s distribution Balls and Bins

• t k ln (k/d)
• s k ln (k/d)

BAD Too much overhead
k ?k ln2(k/d) suffices
GOOD Optimal!
16
Why is s k ln (k/d) optimal?

k bit input

• s balls thrown into k bins
• cant recover input
• ) no green input bits
• Pr no green input bits
• ? k . (1 1/k)s
• ¼ k e-s/k
• Pr failure ? d if s ? k ln k/d

s edges
k bins
s balls
NOTE This line of reasoning is not quite right
for lower bound! Use coupon collector type
argument.
17
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
covered processed ripple
released
STATE
ACTION
Init Release c2, c4, c6
18
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6 covered
a1,a3,a5 processed ripple a1,a3,a5
STATE
ACTION
Process a1
19
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6,c1 covered
a1,a3,a5 processed a1 ripple a3,a5
STATE
ACTION
Process a3
20
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6,c1 covered
a1,a3,a5 processed a1,a3 ripple a5
STATE
ACTION
Process a5
21
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6,c1,c5 covered
a1,a3,a5,a4 processed a1,a3,a5 ripple a4
STATE
ACTION
Process a4
22
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6,c1,c5,c3 covered
a1,a3,a5,a4,a2 processed a1,a3,a5,a4 ripple
a2
STATE
ACTION
Process a2
23
The LT Process
c1
a1
c2
a2
c3
a3
c4
a4
c5
a5
c6
released c2,c4,c6,c1,c5,c3 covered
a1,a3,a5,a4,a2 processed a1,a3,a5,a4,a2 ripp
le
STATE
ACTION
Success!
24
The LT Process Properties

• Corresponds to decoding
• When a code bit cp is released
• The step at which this happensis independent of
  other cqs
• The input bit cp coversis independent of other
  cqs

25
Ripple size

• Desired property of ripple
• Not too large redundant covering
• Not too small might die prematurely
• GOAL Good degree distribution
• Ripple doesnt grow or shrink
• 1 input bit added per step

Why??
26
Degree Distributions

• Degrees of code bits chosen independently
• r(d) Pr degree d
• All 1s distribution r(1) 1, r(d?1) 0
• initial ripple all input bits All-At-Once
  distribution

27
Machinery q(d,L), r(d,L), r(L)

• L unprocessed k, k-1, ,1
• q(d,L) Pr cp is released at L deg(cp)d
• r(d,L) Pr cp is released at L, deg(cp)d
  r(d) q(d,L)
• r(L) Pr cp is released at L åd
  r(d,L)

r(L) controls ripple size
28
q(d,L)
29
Ideal Soliton Distribution, r(.)

• Soliton Wave dispersion balances refraction
• Expected degree ln k
• r(L) 1/k for all L k, , 1

30
Expected Behavior
optimal

• Choose t k
• Exp(s) t Exp(deg) k ln k
• Exp(Initial ripple size) t r(1) 1
• Exp( code bits released per step) t r(L) 1

) Exp(ripple size) 1
31
We expect too much

• What if the ripple vanishes too soon?
• In fact, very likely!
• FIX Robust Soliton Distribution
• Higher initial ripple size ¼ ?k log k/d
• Expected change still 0

32
Robust Soliton Distribution, m(.)

• R c ?k ln k/d
• m(d) (r(d) t(d)) / b where
• t kb

33
Robust Soliton Distribution, m(.)

• t is small
• t kb ? k O(?k ln2 k/d)
• Exp(s) is small
• Exp(s) t åd dm(d) O(k ln k/d)

34
Robust Soliton Distribution, m(.)

• Initial ripple size is not too small
• Exp(Initial ripple size) t m(1)
  ¼ R ¼ ?k ln k/d
• Ripple unlikely to vanish
• Ripple size random walk of length k
• Deviates from its mean by ?k ln k/d with prob
  ? d

35
Robust Release Probability
t r(L) ? L / (L qR) for L R,
const q ? 0
t åLR..2R r(L) ? g R ln R/d for const g gt
0
Proofs on board
36
Pessimistic Filtering

• Let Z ripple size when L bits unprocessed
• Let h Pr released code bit covers input
  bit not in ripple h should be
  around (L Z) / L

If h is lowered to any value ? (L Z)/Lthen
Prsuccess doesnt increase
37
Pessimistic Filtering

• Applying to robust release probability
• t r(L) L/(L qR) turns into t r(L)
  L/(L qR) for worst case analysis
• Will use pessimistic filtering again later

38
Main Theorem Prsuccess 1d

• Idea ripple size is like a random walk of
  length k with mean R ¼ ?k ln k/d
• Initial ripple size qR/2 with prob 1d/3
• Chernoff bound of code bits of deg 1
• Ripple does not vanish for L Rwith prob
  1d/3
• Last R input bits are covered by t(k/R)
  spikewith prob 1d/3

39
Ripple does not vanish for L R

• Let XL code bits released at L
• Exp(XL) L / (L qR)
• Let YL 0-1 random variable with
  Pr YL 0 (L qR) / L
• Let I any end interval of R, , k-1
  starting at L

RipplesizeL qR/2 (åL 2 I XL YL) (kL)
Filtered down init ripplesize
40
Ripple does not vanish for L R

• åL 2 I XL YL (kL)
• åL 2 I (XL YL Exp(XL) YL)
• åL 2 I (Exp(XL) YL Exp(XL) Exp(YL))
  
• åL 2 I (Exp(XL) Exp(YL)) (kL)

qR/4 with prob d/(6k)
0
Pr åL 2 I XL YL (kL) qR/2 d/(3k)
41
Ripple does not vanish for L R

• Recall
• RipplesizeL qR/2 åL 2 I XL YL (kL)
• There are kR intervals I
• Pr Summation qR/2 for some I d/3

0 lt RipplesizeL lt qR with prob 1d/3
Ripple doesnt vanish!
42
Main Theorem Prsuccess 1d

• Idea ripple size is like a random walk of
  length k with mean R ¼ ?k ln k/d
• Initial ripple size qR/2 with prob 1d/3
• Chernoff bound of code bits of deg 1
• Ripple does not vanish for L Rwith prob
  1d/3
• Last R input bits are covered by t(k/R)
  spikewith prob 1d/3

43
Last R input bits are covered

• Recall t åLR..2R r(L) ? g R ln R/d
• By argument similar to Balls and Bins,Pr Last
  R input bits not covered 1 d/3

44
Main Theorem
With Robust Soliton Distribution, the LT Process
succeeds with prob 1 d
t k O(?k ln2 k/d) s O(k ln k/d)