Sparse Solutions of Linear Systems of Equations and Sparse Modeling of Signals and Images

1
Sparse Solutions of Linear Systems of Equations
and Sparse Modeling of Signals and Images

• Alfredo Nava-Tudela
• John J. Benedetto, advisor

2
Outline

• Problem Find sparse solutions of Ax b
• Why do we care?
• An application in signal processing compression
• Working definition of sparse solution
• Why things eventually work, for some cases
• How do we find sparse solutions? OMP algorithm
• Case study What happens when we combine the DCT
  and DWT?
• Conclusions/Recap
• Project timeline

3
Problem
Let A be an n by m matrix with n lt m, and
rank(A)n. We want to solve Ax b, where b is
a data or signal vector, and x is the solution
with the fewest number of non-zero entries
possible, that is, the sparsest
one. Observations - A is underdetermined and,
since rank(A)n, there is an infinite number of
solutions. Good! - How do we find the sparsest
solution? What does this mean in practice? Is
there a unique sparsest solution?
4
But, why do we care?
231 kb, uncompressed, 320x240x3x8 bit
74 kb, compressed 3.241 JPEG
5
But, why do we care?
512 x 512 Pixels, 24-Bit RGB, Size 786 Kbyte
751, 10.6 Kbyte JPEG2000
6
Sparsity equals compression
Both JPEG and JPEG2000 achieve their
compression mainly because at their core one
finds a linear transform (DCT and DWT,
respectively) that reduces the number of non-zero
entries required to represent the data, within an
acceptable error. We can then think of signal
compression in terms of our problem Ax b, x
is sparse, b is dense, store x!
7
Working definitions of sparse

• Convenient to introduce the l0 norm
• x0 k xk ? 0
• (P0) minx x0 subject to Ax - b2 0
• (P0e) minx x0 subject to Ax - b2 lt e
• Observation Solving (P0) is NP-hard, bummer.

8
Some theoretical results
Definition The spark of a matrix A is the
minimum number of linearly dependent columns of
A. We write spark(A) to represent this
number. Theorem If there is a solution x to Ax
b, and x0 lt spark(A) / 2, then x is the
sparsest solution. That is, if y ? x also solves
the equation, then x0 lt y0. Observation
Computing spark(A) is combinatorial, therefore
hard. Alternative?
9
More theoretical results
Definition The mutual coherence of a matrix A is
the number
Lemma spark(A) 11/mu(A). Theorem If x
solves Ax b, and x0 lt (1mu(A)-1)/2, then x
is the sparsest solution, as before. Observation
mu(A) is a lot easier and faster to compute, but
11/mu(A) far worse bound than spark(A), in
general.
10
Finding sparse solutionsOMP
Orthogonal Matching Pursuit algorithm
11
Revisiting compression
Propose to study the compression properties of
the matrix A DCT,DWT and compare it with
the compression properties of DCT or DWT
alone. Study the behavior of OMP for this
problem. Many wavelet options available to try,
e.g., reversible 5/3 or floating-point 9/7
Daubechies, as in JPEG2000. Interested in
compression vs error graph properties.
12
Conclusions/Recapitulation

• Finding sparse solutions to the linear system of
• equations Ax b, when A is an n by m full rank
  matrix and n lt m, is of interest to the signal
  processing community.
• There are simple criteria to assert the
  uniqueness of a given sparse solution.
• There are algorithms to find sparse solutions,
  e.g.,
• OMP and their convergence can be guaranteed when
  there are sufficiently sparse solutions.
• - Studies on the performance of OMP missing when
  A is the concatenation of unitary matrices.

13
Project timeline

• Oct. 15, 2010 Complete project proposal.
• Oct. 18 - Nov. 5 Implement OMP.
• Nov. 8 - Nov. 26 Validate OMP.
• Nov. 29 - Dec. 3 Write mid-year report.
• Dec. 6 - Dec. 10 Prepare mid-year oral
  presentation.
• Some time after that, give mid-year oral
  presentation.
• Jan. 24 - Feb. 11 Testing of OMP. Reproduce
  paper results.
• Jan. 14 - Apr. 8 Testing of OMP on A
  DCT,DWT.
• Apr. 11 - Apr. 22 Write final report.
• Apr. 25 - Apr. 29 Prepare final oral
  presentation.
• Some time after that, give final oral
  presentation.

14
References
A. M. Bruckstein, D. L. Donoho, and M. Elad, From
sparse solutions of systems of equations to
sparse modeling of signals and images, SIAM
Review, 51 (2009), pp. 3481. S. Mallat, A
Wavelet Tour of Signal Processing, Academic
Press, 1998. B. K. Natarajan, Sparse approximate
solutions to linear systems, SIAM Journal on
Computing, 24 (1995), pp. 227-234. G. W.
Stewart, Introduction to Matrix Computations,
Academic Press, 1973. D. S. Taubman and M. W.
Mercellin, JPEG 2000 Image Compression Funda-
mentals, Standards and Practice, Kluwer Academic
Publishers, 2001. G. K. Wallace, The JPEG still
picture compression standard, Communications of
the ACM, 34 (1991), pp. 30-44. http//www.stanfor
d.edu/class/ee398a/handouts/lectures/08-JPEG.pdf