Sparse

1
Sparse Redundant Signal Representation
and
its Role in Image Processing

• Michael Elad
• The Computer Science Department
• The Technion Israel Institute of
  technology
• Haifa 32000, Israel

Mathematics and Image Analysis, MIA'06 Paris,
18-21 September , 2006
2
Todays Talk is About
Sparsity
and
Redundancy

• We will try to show today that
• Sparsity Redundancy can be used to design
  new/renewed powerful signal/image processing
  tools.
• We will present both the theory behind sparsity
  and redundancy, and how those are deployed to
  image processing applications.
• This is an overview lecture, describing the
  recent activity in this field.

3
Agenda

• A Visit to Sparseland
• Motivating Sparsity Overcompleteness
• 2. Problem 1 Transforms Regularizations
• How why should this work?
• 3. Problem 2 What About D?
• The quest for the origin of signals
• 4. Problem 3 Applications
• Image filling in, denoising, separation,
  compression,

Welcome to
Sparseland
4
Generating Signals in Sparseland
M?
5
Sparseland Signals Are Special

• Simple Every generated signal is built as a
  linear combination of few atoms from our
  dictionary D
• Rich A general model the obtained signals are a
  special type mixture-of-Gaussians (or Laplacians).

6
Transforms in Sparseland ?

• We desire simplicity, independence, and
  expressiveness.

7
So, In Order to Transform

• Among all (infinitely many) possible solutions we
  want the sparsest !!

8
Measure of Sparsity?
9
Signals Transform in Sparseland

• How effective are those ways?

• How would we get D?

10
Inverse Problems in Sparseland ?

• How about find the a that generated the x
  again?

11
Inverse Problems in Sparseland ?
12
Back Home Any Lessons?
Several recent trends worth looking at

• JPEG to JPEG2000 - From (L2-norm) KLT to wavelet
  and non-linear approximation

• From Wiener to robust restoration From L2-norm
  (Fourier) to L1. (e.g., TV, Beltrami, wavelet
  shrinkage )

• From unitary to richer representations Frames,
  shift-invariance, steerable wavelet, contourlet,
  curvelet

• Approximation theory Non-linear approximation

• ICA and related models

13
To Summarize so far
The Sparseland model for signals is very
interesting
We do! it is relevant to us
We need to answer the 4 questions posed, and then
show that all this works well in practice
14
Agenda
1. A Visit to Sparseland Motivating Sparsity
Overcompleteness 2. Problem 1 Transforms
Regularizations How why should this work?
3. Problem 2 What About D? The quest for the
origin of signals 4. Problem 3
Applications Image filling in, denoising,
separation, compression,
15
Lets Start with the Transform
16
Question 1 Uniqueness?
17
Matrix Spark
Rank 4
18
Uniqueness Rule
19
Question 2 Practical P0 Solver?
M?
20
Matching Pursuit (MP)
Mallat Zhang (1993)

• The MP is a greedy algorithm that finds one atom
  at a time.

• Step 1 find the one atom that best matches the
  signal.

• Next steps given the previously found atoms,
  find the next one to best fit

• The Orthogonal MP (OMP) is an improved version
  that re-evaluates the coefficients after each
  round.

21
Basis Pursuit (BP)
Chen, Donoho, Saunders (1995)

• The newly defined problem is convex (linear
  programming).
• Very efficient solvers can be deployed
• Interior point methods Chen, Donoho, Saunders
  (95) ,
• Sequential shrinkage for union of ortho-bases
  Bruce et.al. (98),
• Iterated shrinkage Figuerido Nowak (03),
  Daubechies, Defrise, Demole (04), E. (05),
  E., Matalon, Zibulevsky (06).

22
Question 3 Approx. Quality?
M?
23
Evaluating the Spark

• Compute

Assume normalized columns

• The Mutual Coherence M is the largest
  off-diagonal entry in absolute value.

24
BP and MP Equivalence
Equivalence
Given a signal x with a representation
, Assuming that , BP and
MP are Guaranteed to find the sparsest solution.

Donoho E. (02) Gribonval Nielsen
(03) Tropp (03) Temlyakov (03)

• MP and BP are different in general (hard to say
  which is better).
• The above result corresponds to the worst-case.
• Average performance results are available too,
  showing much better bounds Donoho (04), Candes
  et.al. (04), Tanner et.al. (05), Tropp et.al.
  (06), Tropp (06).

25
What About Inverse Problems?

• We had similar questions regarding uniqueness,
  practical solvers, and their efficiency.
• It turns out that similar answers are applicable
  here due to several recent works Donoho, E. and
  Temlyakov (04), Tropp (04), Fuchs (04),
  Gribonval et. al. (05).

26
To Summarize so far
The Sparseland model is relevant to us. We can
design transforms and priors based on it
Use pursuit Algorithms
A sequence of works during the past 3-4 years
gives theoretic justifications for these tools
behavior

1. How shall we find D?
2. Will this work for applications? Which?

27
Agenda
1. A Visit to Sparseland Motivating Sparsity
Overcompleteness 2. Problem 1 Transforms
Regularizations How why should this work?
3. Problem 2 What About D? The quest for the
origin of signals 4. Problem 3
Applications Image filling in, denoising,
separation, compression,
28
Problem Setting
M?
Multiply by D
Given these P examples and a fixed size N?K
dictionary D, how would we find D?
29
Uniqueness?
If is rich enough and if then D is
unique.
Uniqueness
Aharon, E., Bruckstein (05)
Comments

• This result is proved constructively, but the
  number of examples needed to pull this off is
  huge we will show a far better method next.

• A parallel result that takes into account noise
  is yet to be constructed.

30
Practical Approach Objective
Field Olshausen (96) Engan et. al.
(99) Lewicki Sejnowski (00) Cotter et. al.
(03) Gribonval et. al. (04)
(n,K,L are assumed known, D has norm. columns)
31
KMeans For Clustering
Clustering An extreme sparse coding
32
The KSVD Algorithm General
T
Aharon, E., Bruckstein (04)
33
KSVD Sparse Coding Stage
T
Pursuit Problem !!!
34
KSVD Dictionary Update Stage
D
35
KSVD Dictionary Update Stage
D
ResidualE
36
KSVD A Synthetic Experiment
37
To Summarize so far
The Sparseland model for signals is relevant to
us. In order to use it effectively we need to
know D
Use the K-SVD algorithm
We have shown how to practically train D using
the K-SVD
Show that all the above can be deployed to
applications
38
Agenda
1. A Visit to Sparseland Motivating Sparsity
Overcompleteness 2. Problem 1 Transforms
Regularizations How why should this work?
3. Problem 2 What About D? The quest for the
origin of signals 4. Problem 3
Applications Image filling in, denoising,
separation, compression,
39
Application 1 Image Inpainting

• Assume the signal x has been created
  by xDa0 with very
  sparse a0.
• Missing values in x imply
  missing
  rows in this linear
  system.
• By removing these rows, we get .
• Now solve
• If a0 was sparse enough, it will be the solution
  of the above problem! Thus, computing Da0
  recovers x perfectly.

40
Application 1 Side Note

• Compressed Sensing is leaning on the very same
  principal, leading to alternative sampling
  theorems.

• Assume the signal x has been created by xDa0
  with very sparse a0.

• Multiply this set of equations by the matrix Q
  which reduces the number of rows.

• The new, smaller, system of equations is

• If a0 was sparse enough, it will be the sparsest
  solution of the new system, thus, computing Da0
  recovers x perfectly.
• Compressed sensing focuses on conditions for this
  to happen, guaranteeing such recovery.

41
Application 1 The Practice

• Given a noisy image y, we can clean it using the
  Maximum A-posteriori Probability estimator by
  Chen, Donoho, Saunders (95).

• What if some of the pixels in that image are
  missing (filled with zeros)? Define a mask
  operator as the diagonal matrix W, and now solve
  instead

• When handling general images, there is a need to
  concatenate two dictionaries to get an effective
  treatment of both texture and cartoon contents
  This leads to separation E., Starck, Donoho
  (05).

42
Inpainting Results
Predetermined dictionary
Curvelet (cartoon) Overlapped DCT (texture)
43
Inpainting Results
More about this application will be given in
Jean-Luc Starck talk that follows.
44
Application 2 Image Denoising

• Given a noisy image y, we have already mentioned
  the ability to clean it by solving

• When using the K-SVD, it cannot train a
  dictionary for large support images How do we
  go from local treatment of patches to a global
  prior?

• The solution Force shift-invariant sparsity - on
  each patch of size N-by-N (e.g., N8) in the
  image, including overlaps.

45
Application 2 Image Denoising
46
Application 2 Image Denoising
D?

• The dictionary (and thus the image prior) is
  trained on the corrupted itself!
• This leads to an elegant fusion of the K-SVD and
  the denoising tasks.

xy and D known
x and ?ij known
D and ?ij known
47
Application 2 The Algorithm
Sparse-code every patch of 8-by-8 pixels
D
Update the dictionary based on the above codes
The computational cost of this algorithm is
mostly due to the OMP steps done on each image
patch - O(N2LKIterations) per pixel.
Compute the output image x
48
Denoising Results
The results of this algorithm tested over a
corpus of images and with various noise powers
compete favorably with the state-of-the-art - the
GSMsteerable wavelets denoising algorithm by
Portilla, Strela, Wainwright, Simoncelli
(03), giving 1dB better
results on average.
49
Application 3 Compression

• The problem Compressing photo-ID images.
• General purpose methods (JPEG, JPEG2000)
  do not take
  into account the specific family.
• By adapting to the image-content (PCA/K-SVD),
  better results
  could be obtained.
• For these techniques to operate well, train
  dictionaries
  locally (per patch) using a
  training
  set of images is required.
• In PCA, only the (quantized) coefficients are
  stored, whereas
  the K-SVD requires storage of the indices
  as well.
• Geometric alignment of the image is very helpful
  and
  should be done.

50
Application 3 The Algorithm
Detect main features and warp the images to a
common reference (20 parameters)
Training set (2500 images)
51
Compression Results
Results for 820 Bytes per each file
52
Compression Results
Results for 550 Bytes per each file
53
Compression Results
Results for 400 Bytes per each file
54
Today We Have Discussed
1. A Visit to Sparseland Motivating Sparsity
Overcompleteness 2. Problem 1 Transforms
Regularizations How why should this work?
3. Problem 2 What About D? The quest for the
origin of signals 4. Problem 3
Applications Image filling in, denoising,
separation, compression,
55
Summary
Sparsity and Over- completeness are important
ideas. Can be used in designing better tools in
signal/image processing

• This is an on-going work
• Deepening our theoretical
• understanding,
• Speedup of pursuit methods,
• Training the dictionary,
• Demonstrating applications,

Future transforms and regularizations should be
data-driven, non-linear, overcomplete, and
promoting sparsity.
56
Thank You for Your Time
More information, including these slides, can be
found in my web-page http//www.cs.technion.ac.i
l/elad
THE END !!