Capacity Upper Bounds for Deletion Channels

1
Capacity Upper Bounds for Deletion Channels

• Suhas Diggavi
• Michael Mitzenmacher
• Henry Pfister

2
The Most Basic Channels

• Binary erasure channel.
• Each bit is replaced by a ? with probability p.
• Binary symmetric channel.
• Each bit flipped with probability p.
• Binary deletion channel.
• Each bit deleted with probability p.

3
The Most Basic Channels

• Binary erasure channel.
• Each bit is replaced by a ? with probability p.
• Very well understood.
• Binary symmetric channel.
• Each bit flipped with probability p.
• Very well understood.
• Binary deletion channel.
• Each bit deleted with probability p.
• We dont even know the capacity!!!

4
The Challenge

• Would like a single-letter characterization of
  capacity.
• Or tight upper/lower bounds.
• Or effective means of calculating capacity.
• Such characterizations are difficult.
• Deletion channels are channels with memory.

5
Recent Progress

• Chain of results giving better lower bounds.
• Shannon-style arguments.
• Diggavi/Grossglauser, Drinea/Mitzenmacher,
  Drinea/Kirsch.
• Global result capacity is at least (1 p)/9.
• But essentially no work on upper bounds.
• Ullmans bound zero-error decoding for
  worst-case synchronization errors. (Does not
  apply.)
• Trivial bound of (1 p).
• Lower bound progress motivates need for progress
  in the other direction.
• How close are we getting to capacity???

6
Our Results

• An upper bound approach using side information
  that gives numerical bounds.
• An upper bound approach using side information
  that gives asymptotic behavior as fraction of
  deletions p goes to 1, for a bound of c(1 p).

7
Upper Bound via Run Lengths

• Input can be thought of as runs of maximally
  contiguous 0s/1s.
• Side information Suppose an undeletable marker
  inserted every time a complete run is deleted.
• Can only increase capacity.
• Equivalent to a memoryless channel where symbols
  are run lengths.

0000110111000110000.
8
Example
0000110111000110000.
Sent
00111100000.
Received
4 2 1 3 3 2 4.
Sent
2 2 0 2 3 0 2.
Received
9
Capacity Per Unit Cost

• Associate cost of x with run of length x at
  input.
• Capacity of binary channel with side info
  Capacity per unit cost of run length channel.
• Latter can be upper bounded numerically using
  Kuhn-Tucker conditions.
• Challenging because of infinite alphabet.

10
Upper Bound Statement

• For channel defined by pY X and any output
  distribution qY let
• Then for any non-negative cost function c(x), the
  capacity per unit cost C satisfies
• Abdel-Ghaffar 1993

11
Upper Bound Calculation

• Problem Optimization over infinite alphabet.
• Solution Finitize the problem.
• Find optimal input/output distribution for
  truncated alphabet.
• Replace tail of finite output distribution with
  geometric distribution.
• To allow analytic bound on
  for large x.
• Bound performance of resulting distribution.
• Optimize over truncated alphabet.

12
Bounds Achieved
13
Asymptotic Result

• Motivation
• Previous upper bound approach breaks down for
  large p.
• Not surprising large p implies more completely
  deleted runs, so more side information released.
• Want to find limits of possible global results.
• The (1 p)/9 lower bound argument seems tightest
  as p approaches 1.
• Can we obtain an asymptotic c(1 p) upper bound?
  
• Build upon insights from global lower bound.

14
Upper Bound via Markers

• Suppose an undeletable marker is inserted every
  1/(1 p) bits in the transmission.
• Channel now memoryless.
• Input symbols 1/(1 p) bits.
• Output symbols random subsequence, with
  expected length 1.
• Capacity should scale with (1 p).
• Capacity bound
• How can we optimize over such a large dimensional
  space?
• Symbols are big.

15
Upper Bound Calculation Step 1, Output

• Problem Optimization over all output symbols.
• Potentially infinitely many bit strings.
• Solution Finitize the problem.
• At receiver, number of bits between markers
  converges to Poisson distribution.
• For upper bounds, assume that if k gt 6 bits
  received, then receiver obtained k bits of
  information.
• Only affects bounds by a few percent.
• Only need to consider outputs of 6 or fewer bits.

16
Upper Bound Calculation Step 2, Input

• Problem Optimization over input strings.
• Sequence of 1/(1 p) bits. Potentially infinite
  alphabet.
• Solution Finitize the problem.
• Key Lemma if only considering up to 6 bits at
  output, need only consider sequences of up to 6
  runs at input.
• Same output distribution achieved by convex
  combination.
• Upper bound achieved by optimizing over large
  number of finite-dimensional vectors representing
  up to 6 runs.
• Heuristic/computational approach.

17
Bounds Achieved

• As p goes to 1, cannot obtain capacity better
  than 0.7918(1 p).
• Gap between (asymptotic) upper/lower bounds now
  roughly (1 p)/9 and 4(1 p)/5.
• Room for improvement, probably both sides.

18
Conclusions and Open Questions

• What are the limitations of such side information
  arguments?
• Are novel upper bound techniques required for
  these channels?
• Is there a more purely information theoretic
  approach?
• Can we characterize optimal input/output
  distributions?
• Heuristic/other approaches to guide theory?
• How tight can we make upper/lower bounds?
• What is the right answer?
• Extend upper bounds for insertion channels?
• Many, many others