Sparse Recovery Using Sparse Random Matrices

1
Sparse Recovery Using Sparse (Random) Matrices

• Piotr Indyk
• MIT

Joint work with Radu Berinde, Anna Gilbert,
Howard Karloff, Martin Strauss and Milan Ruzic
2
Linear Compression(a.k.a. linear sketching,
compressed sensing)

• Setup
• Data/signal in n-dimensional space x
• E.g., x is an 1000x1000 image ?
  n1000,000
• Goal compress x into a sketch Ax ,
• where A is a carefully designed m x n
  matrix, m ltlt n
• Requirements
• Plan A want to recover x from Ax
• Impossible undetermined system of equations
• Plan B want to recover an approximation x of
  x
• Sparsity parameter k
• Want x such that x-xp? C(k) x-xq
  ( lp/lq guarantee )
• over all x that are k-sparse (at most k
  non-zero entries)
• The best x contains k coordinates of x with the
  largest abs value
• Want
• Good compression (small m)
• Efficient algorithms for encoding and recovery
• Why linear compression ?

3
Applications
4
Application I Monitoring Network Traffic

• Router routs packets
• (many packets)
• Where do they come from ?
• Where do they go to ?
• Ideally, would like to maintain a traffic
• matrix x.,.
• Easy to update given a (src,dst) packet,
  increment xsrc,dst
• Requires way too much space!
• (232 x 232 entries)
• Need to compress x, increment easily
• Using linear compression we can
• Maintain sketch Ax under increments to x, since
  A(x?) Ax A?
• Recover x from Ax

destination
source
x
5
Other applications

• Single pixel camera
• Wakin, Laska, Duarte, Baron, Sarvotham,
  Takhar, Kelly, Baraniuk06
• Microarray Experiments/Pooling Kainkaryam,
  Gilbert, Shearer, Woolf, Hassibi et al,
  Dai-Sheikh, Milenkovic, Baraniuk

6
Known constructionsalgorithms
7
Constructing matrix A

• Choose encoding matrix A at random
• (the algorithms for recovering x are more
  complex)
• Sparse matrices
• Data stream algorithms
• Coding theory (LDPCs)
• Dense matrices
• Compressed sensing
• Complexity theory (Fourier)
• Traditional tradeoffs
• Sparse computationally more efficient, explicit
• Dense shorter sketches
• Goals unify, find the best of all worlds

8
Result Table
Scale
Excellent
Very Good
Good
Fair

• Legend
• ndimension of x
• mdimension of Ax
• ksparsity of x
• T iterations
• Approx guarantee
• l2/l2 x-x2 ? Cx-x2
• l2/l1 x-x2 ? Cx-x1/k1/2
• l1/l1 x-x1 ? Cx-x1

Caveats (1) only results for general vectors x
are shown (2) all bounds up to O() factors (3)
specific matrix type often matters (Fourier,
sparse, etc) (4) ignore universality,
explicitness, etc (5) most dominated algorithms
not shown
9
Techniques
10
dense vs. sparse

• Restricted Isometry Property (RIP) - key property
  of a dense matrix A
• x is k-sparse ? x2? Ax2 ? C x2
• Holds w.h.p. for
• Random Gaussian/Bernoulli mO(k log (n/k))
• Random Fourier mO(k logO(1) n)
• Consider random m x n 0-1 matrices with d ones
  per column
• Do they satisfy RIP ?
• No, unless m?(k2) Chandar07
• However, they can satisfy the following RIP-1
  property Berinde-Gilbert-Indyk-Karloff-Strauss08
  
• x is k-sparse ? d (1-?) x1? Ax1 ?
  dx1
• Sufficient (and necessary) condition the
  underlying graph is a
• ( k, d(1-?/2) )-expander

11
Expanders

• A bipartite graph is a (k,d(1-?))-expander if for
  any left set S, Sk, we have N(S)(1-?)d S
• Constructions
• Randomized mO(k log (n/k))
• Explicit mk quasipolylog n
• Plenty of applications in computer science,
  coding theory etc.
• In particular, LDPC-like techniques yield good
  algorithms for exactly k-sparse vectors x
• Xu-Hassibi07, Indyk08, Jafarpour-Xu-Hassibi-Ca
  lderbank08

N(S)
d
S
m
n
12
dense vs. sparse

• Instead of RIP in the L2 norm, we have RIP in the
  L1 norm
• Suffices for these results
• Main drawback l1/l1 guarantee
• Better approx. guarantee with same time and
  sketch length
• Other sparse matrix schemes, for (almost)
  k-sparse vectors
• LDPC-like Xu-Hassibi07, Indyk08,
  Jafarpour-Xu-Hassibi-Calderbank08
• L1 minimization Wang-Wainwright-Ramchandran08
  
• Message passing Sarvotham-Baron-Baraniuk06,08,
  Lu-Montanari-Prabhakar08

?
13
Algorithms/Proofs
14
Proof d(1-?/2)-expansion ? RIP-1

• Want to show that for any k-sparse x we have
• d (1-?) x1? Ax1 ? dx1
• RHS inequality holds for any x
• LHS inequality
• W.l.o.g. assume
• x1 xk xk1 xn0
• Consider the edges e(i,j) in a lexicographic
  order
• For each edge e(i,j) define r(e) s.t.
• r(e)-1 if there exists an edge (i,j)lt(i,j)
• r(e)1 if there is no such edge
• Claim Ax1 ?e(i,j) xire

15
Proof d(1-?/2)-expansion ? RIP-1 (ctd)

• Need to lower-bound
• ?e zere
• where z(i,j)xi
• Let Rb the sequence of the first bd res
• From graph expansion, Rb contains at most ?/2 bd
  -1s
• (for b1, it contains no -1s)
• The sequence of res that minimizes ?ezere is
• 1,1,,1,-1,..,-1 , 1,,1,
• d ?/2 d
  (1-?/2)d
• Thus
• ?e zere (1-?)?e ze (1-?) dx1

d
16
A satisfies RIP-1 ? LP works Berinde-Gilbert-Indy
k-Karloff-Strauss08

• Compute a vector x such that AxAx and x1
  minimal
• Then we have, over all k-sparse x
• x-x1 C minx x-x1
• C?2 as the expansion parameter ??0
• Can be extended to the case when Ax is noisy

17
A satisfies RIP-1 ? Sparse Matching Pursuit
worksBerinde-Indyk-Ruzic08

• Algorithm
• x0
• Repeat T times
• Compute cAx-Ax A(x-x)
• Compute ? such that ?i is the median of its
  neighbors in c
• Sparsify ?
• (set all but 2k largest entries of ? to 0)
• xx?
• Sparsify x
• (set all but k largest entries of x to 0)
• After Tlog() steps we have
• x-x1 C min k-sparse x x-x1

A
c
?
18
Experiments

• Probability of recovery of random k-sparse
  1/-1 signals
• from m measurements
• Sparse matrices with d10 1s per column
• Signal length n20,000

SMP
LP
19
Conclusions

• Sparse approximation using sparse matrices
• State of the art can do 2 out of 3
• Near-linear encoding/decoding
• O(k log (n/k)) measurements
• Approximation guarantee with respect to L2/L1
  norm
• Open problems
• 3 out of 3 ?
• Explicit constructions ?

This talk
20
Resources

• References
• R. Berinde, A. Gilbert, P. Indyk, H. Karloff,
  M. Strauss, Combining geometry and
  combinatorics a unified approach to sparse
  signal recovery, Allerton, 2008.
• R. Berinde, P. Indyk, M. Ruzic, Practical
  Near-Optimal Sparse Recovery in the L1 norm,
  Allerton, 2008.
• R. Berinde, P. Indyk, Sparse Recovery Using
  Sparse Random Matrices, 2008.
• P. Indyk, M. Ruzic, Near-Optimal Sparse Recovery
  in the L1 norm, FOCS, 2008.

21
Running times