Compressed Sensing

1
Compressed Sensing

• Trac D. Tran
• ECE Department
• The Johns Hopkins University
• Baltimore, MD 21218

2
Outline

• Compressed Sensing Quick Overview
• Motivations. Toy Examples
• Incoherent Bases and Restrictive Isometry
  Property
• Decoding Strategy
• L0 versus L1 versus L2
• Basis Pursuit and Matching Pursuit
• Examples of Compressed Sensing Applications
• One-pixel Camera
• 2D Separable Measurement Ensemble (SME)
• Face/Speech Recognition
• MR Imaging
• Distributed compressed video sensing (DISCOS)
• Layered compressed sensing for robust video
  transmission

3
Compressed Sensing History

• Emmanuel Candès and Terence Tao, Decoding by
  linear programming IEEE Trans. on Information
  Theory, 51(12), pp. 4203 - 4215, December 2005
• Emmanuel Candès, Justin Romberg, and Terence
  Tao, Robust uncertainty principles Exact signal
  reconstruction from highly incomplete frequency
  information, IEEE Trans. on Information Theory,
  52(2) pp. 489 - 509, Feb. 2006.
• David Donoho, Compressed sensing, IEEE Trans.
  on Information Theory, 52(4), pp. 1289 - 1306,
  Apr. 2006.
• Emmanuel Candès and Michael Wakin, An
  introduction to compressive sampling, IEEE
  Signal Processing Magazine, 25(2), pp. 21 - 30,
  Mar. 2008.

4
Traditional Data Acquisition Sampling

• Shannon Sampling Theorem

In order for a band-limited signal x(t) to be
reconstructed perfectly, it must be sampled at
rate
5
Traditional Compression Paradigm

• Sample first and then worry about compression
  later!

6
Sparse Signals
basis functions
largest coefficients
transform coefficients

• Digital signals in practice are often sparse
• Audio MP3, AAC 101 compression
• Images JPEG, JPEG2000 201 compression
• Video sequences MPEG2, MPEG4 401 compression

7
Sparse Signals II
basis functions
N-pixel image
transform coefficients
8
Definition Notation

• N length of signal x
• K the sparsity level of x or x is called
  K-sparse
• M the number of measurements (samples) taken at
  the encoder

9
Compressed Sensing Framework

• Encoding obtain M measurements y from linear
  projection onto an incoherent basis

has only K nonzero entries
Compressed measurements

• Decoding reconstruct x from measurements y via
  nonlinear optimization with sparsity prior

10
At Encoder Signal Sensing

• y is not sparse, looks iid

• Random projection works well!

• Sensing sparsifying matrix must be incoherent

• Each measurement y contains a little information
  of each sample of x

11
At Decoder Signal Reconstruction

• Recover x from the set of measurements y
• Without the sparseness assumption, the problem is
  ill-posed
• With sparseness assumption, the L0-norm
  minimization problem is well-posed but
  computationally intractable
• With sparseness assumption, the L1-norm
  minimization can be solved via linear programming
  Basis Pursuit!

12
Incoherent Bases Definition
Suppose signal is sparse in a orthonomal
transform domain
Take K measurements from an orthonormal sensing
matrix
Definition Coherence between and
T
With ,
13
Incoherent Bases Properties
Bound of coherence
When is small, we call 2 bases are
incoherent
Intuition when 2 bases are incoherent, all
entries of matrix are spread out
each measurement will contains more information
of signal we hope to have small
Some pair of incoherent bases
DFT and identity matrix
Gaussian (Bernoulli) matrix and any other basis
14
Universality of Incoherent Bases

• Random Gaussian white noise basis is incoherent
  with any fixed orthonormal basis with high
  probability
• If the signal is sparse in frequency, the
  sparsifying matrix is
• Product of is still Gaussian white noise!

15
Restricted Isometry Property
non-zero entries

• Sufficient condition for exact recovery All
  sub-matrices composed of columns are
  nearly orthogonal

16
Unique Exact Solution?
Suppose we can find a subset S columms of matrix
A with SK such that columns of
are dependent . It means
Decompose with each has
sparsity K
2 sparse signal produce the same measurements
No way to find the exact solution
A sufficient condition must
be invertible
17
How Many Measurements Are Enough?
Theorem (Candes, Romberg and Tao) Suppose x
has support on T , M rows of matrix F is
selected uniformly at random from N rows of DFT
matrix N x N, then if M obeying
Minimize L1 will recover x exactly with
extremely high probability
In practice
is enough to perfectly recover
18
L0- and L1-norm Reconstruction
- norm reconstruction take advantage of
sparsity prior
We find the sparsest solution
Problem Combinatorial searching
Exhaustive computation
- norm reconstruction Compressed sensing
framework
This is a convex optimization problem
Using linear programming to solve
Also can find the sparsest which
turns out to be the exact solution
- 18 -
19
L1-Minimization

• Problem
• Let
• Standard LP
• Many available techniques
• Simplex, primal-dual interior-point, log-barrier

20
L2-norm Reconstruction
- norm reconstruction classical approach
with smallest energy
We find
Closed-form solution
Unfortunately, this method almost never find the
sparsest and correct answer
21
Why Is L1 Better Than L2?
Bad point
circle at a non-sparse point
The line
intersect
The line
intersect
diamond at the sparse point
Unique and exact solution
- 21 -
22
CS Reconstruction Matching Pursuit

• Problem
• Basis Pursuit
• Greedy Pursuit Iterative Algorithms
• At each iteration, try to identify columns of A
  (atoms) that are associated with non-zero entries
  of

23
Matching Pursuit
MP At each iteration, MP attempts to identify
the most significant atom. After K iteration, MP
will hopefully identify the signal!

• , set residual vector ,
  selected index set
• Find index yielding the maximal correlation
  with the residue
• Augment selected index set
• Update the residue
• , and stop when

t K
24
Orthogonal Matching Pursuit
OMP guarantees that the residue is orthogonal to
all previously chosen atoms ? no atom will
be selected twice!

• , set residual vector , index
  set
• Find index that yields the maximal
  correlation with residue
• Augment
• Find new signal estimate by solving
• Set the new residual
• , and stop when

t K
25
Subspace Pursuit
SP pursues the entire subspace that the signal
lives in at every iteration steps and adds a
backtracking mechanism!

• Initialization
• Selected set
• Signal estimate
• Residue
• , go to Step 2 stop when residue
  energy does not decrease anymore

26
CS Applications

• Practical Compressed Sensing or Sparse Signal
  Processing
• One-pixel camera
• 2D separable measurement ensemble for image/video
• Face recognition
• Speech recognition
• Distributed compressed video sensing
• Layered compressed-sensing robust video
  transmission
• Video denoising
• Video super-resolution
• Multiple description coding
• MRI application

27
One-Pixel Compressed Sensing Camera

• Courtesy of Richard Baraniuk Kevin Kelly _at_ Rice

28
Common Sensing Limitations

• Treat every source signal as 1D signal perform
  sensing operation on a vectorized signal
• Increase significant complexity at both encoder
  and decoder
• Inappropriate for some compressive imaging
  applications such as Compressive Image Sensors
• Physical structure of image sensor arrays is 2D
• Costly implementation of dense sensing matrices
  due to wide dynamical range issue
• Block-diagonal sensing matrices results in
  low-performance due to incoherence degradation
  issue

29
2D Separable Measurement Ensembles

• Y D1 F1 P1 S1 X S2 P2 F2 D2

Randomly Flipping Sign of Rows and Columns
Randomly subsampling rows
Randomly subsampling columns
Randomly Permuting Rows and Columns
WHT Block-Diagonal Fast Transform
Entry 0
Entry 1
Entry -1
30
2D Separable Measurement Ensembles

• In algorithm
• Randomly flip sign of rows columns of a source
  image
• Randomly permute rows and columns of sign-flipped
  image
• Transform the randomized image by Walsh-Hadamard
  block-diagonal matrix
• Randomly subsample rows columns of the
  transform image
• All operations are performed on rows and columns
  of a source image, separately

31
Underlying Principles

• Issue Smooth images tend to be highly coherent
  with fixed linear transforms
• Solution Randomly flip sign of rows and columns
  of a source image to increase its incoherence

Flipping sign of rows
Flipping sign of columns
32
Underlying Principles

• Issue Block-diagonal sensing matrices tend to be
  highly coherent with a source signal
• Solution Randomly permute row and column of a
  block-diagonal sensing matrix to increase its
  incoherence

Permute columns of block-diagonal matrix
Permute rows of block-diagonal matrix
33
Underlying Principles

• Preprocess a source image before subsampling its
  rows and columns
• Spread out the energy along rows and columns
• Guarantee energy preservation of a subset of
  measurements (submatrix) if coupling with a
  suitable scale factor (with high probability)

Preprocessing
34
Simulation Results
(a) SRM PSNR 29.4 dB

• Performance Comparison 512x512 Lena
• 1D Non-separable ME (SRM Do)
• 2D Separable ME using block size 32x32

(b) 2D-SME, PSNR 28 dB
Reconstruction algorithm GPSR Figueiredo
Reconstruction from 25 measurement
35
Simulation Results
(a) SRM PSNR 29.3 dB

• Performance Comparison 1024x1024 Man
• 1D Non-separable ME (SRM Do)
• 2D Separable ME using block size 32x32

(b) 2D-SME, PSNR 28 dB
Reconstruction algorithm GPSR Figueiredo
Reconstruction from 35 measurement
36
Application in Face Recognition

• Face-subspace model faces under varying lighting
  and expression lie on a subspace
• A new test sample y of object i approximately
  lies in the linear span of the training samples
  associated with i
• The test sample y is then a sparse linear
  combination of all training samples
• Sparse representation
• encodes the membership i of y

John Wright, Allen Y. Yang, Arvind Ganesh, S.
Shankar Sastry, and Yi Ma, Robust Face
Recognition via Sparse Representation, IEEE
Trans. PAMI, Feb. 2009
37
Sparse Classification

• Classification problem can be solved by
• If the solution is sparse enough
• Test data y is classified based on the residual
• where the only nonzero entries in
  are the entries in x that are associated with
  class i

38
Example I
Coefficients
Residual Energy
39
Robustness to Corruption

• Test image y can be corrupted or occluded
• If the occlusion e covers less than 50 of the
  image, the sparsest solution to y B w is the
  true generator w0
• If w0 is sparse enough, can be solved by
• Classified based on the residual

40
Example II Set-Up

• Top left face image is occluded by a disguise
• Bottom left 50 of random pixels replaced by
  random values
• Right training images

41
Example II Result

• Test images sum of a sparse linear combination
  of the training images and a sparse error due to
  occlusion or corruption

42
Compressed Sensing in Medical Imaging

• Goal so far
• Achieve faster MR imaging while maintaining
  reconstruction quality
• Methods
• under sample discrete Fourier space using some
  pseudo-random patterns
• then reconstruct using L1-minimization (Lustig)
  or homotopic L0-minimization (Trzasko)

43
Sparsity of MR images

• Brain MR images, cardiology dynamic MR images
• Sparse in Wavelet, DCT domains
• Not sparse in spatial domain, or
  finite-difference domain
• Can be reconstructed with good quality using
  5-10 of coefficients
• Angiogram images
• Space in finite-difference domain and spatial
  domain
• (edges of blood vessels occupy only 5 in space)
• allow very good Compressed Sensing
  performance

44
Sampling Methods

• Using smooth k-space trajectories
• Cartesian scanning
• Radial scanning
• Spiral scanning
• Fully sampling in each read-out
• Under sampling by
• Cartesian grid under sampling in phase encoding
  (uniform, non-uniform)
• Radial angular under-sampling (using less
  angles)
• Spiral using less spirals and randomly perturb
  spiral trajectories

45
Sampling Patterns (rect)
Varying density
- Under-sampling in phase-encoding (v) -
Full-sampling in frequency-encoding (u)
Uniform density
46
Sampling patterns and TPSF analysis
47
Transform PSF analysis

• PSF is a natural tool to measure incoherence
• Images of interest are not sparse in space domain
  gt define Transform PSF
• where ? is an orthogonal sparsifying transform
  (TV, wavelet, DCT) Fu under-sampled Fourier
  operator
• TPSF measures how a single transform coef of the
  underlying object influences other transform
  coefs of the measured under-sampled object.

48
Sampling Patterns Spiral Radial
Spiral scanning uniform density, varying
density, and perturbed spiral trajectories
New algorithm (FOCUSS) allows reconstruction
without angular aliasing artifacts
49
Reconstruction Methods

• Lustigs L1-minimization with non-linear
  Conjugate Gradient method
• Trzaskos homotopic L0-minimization

50
Reconstruction Results (2DFT)
Multi-slice 2DFT fast spin echo CS at 2.4
acceleration.
51
Results 3DFT Contrast-enhanced 3D angiography
reconstruction results as a function of
acceleration. Left Column Acceleration by LR.
Note the diffused boundaries with acceleration.
Middle Column ZF-w/dc reconstruction. Note the
increase of apparent noise with acceleration.
Right Column CS reconstruction with TV penalty
from randomly under-sampled k-space
52
1 3
2 4
Results Radial scan, FOCUSS reconstruction Recon
struction results from full-scan with uniform
angular sampling between 0?360?. 1st row
Reference reconstruction from 190 views. 2nd
row Reconstruction results from 51 views using
LINFBP. 3rd row Reconstruction results from 51
views using CG-ALONE. 4th row Reconstruction
results from 51 views using PR-FOCUSS
53
Results spiral (a) Sagittal T2-weighted image
of the spine, (b) simulated k-space trajectory
(multishot Cartesian spiral, 83 under-sampling),
(c) minimum energy solution via zero-filling,
(d) reconstruction by L1 minimization, (e)
reconstruction by homotopic L0 minimization using
?(?u, ? ) ?u/ (?u ?), (f) line profile
across C6, (g-j) enlargements of (a,c-e),
respectively.
54
Homotopic L0-Minimization

• Originally we want to minimize

• This is not a convex optimization problem gt
  approximate L0-norm by a semimetric function ?(.)

55
Inter-frameVideo Coding

• Examples Video Compression Standards MPEG/H.26x
• High complexity at Encoder Low complexity at
  Decoder

56
Principle of Block Motion Estimation

• Partition current video frame into small
  non-overlapped blocks called macro-blocks (MB)
• For each block, within a search window, find the
  motion vector (displacement) that minimizes a
  pre-defined mismatch error
• For each block, motion vector and prediction
  error (residue) are encoded

MV
Search Window
Reference Frame
Current Frame
57
BME/BMC Example I
Previous Frame
Current Frame
Motion Vector Field
Frame Difference
Motion-Compensated Difference
58
BME/BMC Example II
Previous Frame
Current Frame
Motion Vector Field
Frame Difference
Motion-Compensated Difference
59
Distributed Video Coding (DVC)

• Examples PRISM at Berkeley, Turbo Coding with a
  feedback channel at Stanford, etc.
• All based on Wyner-Ziv coding techniques
• Low complexity at Encoder High complexity at
  Decoder

60
Distributed Video Coding (DVC)

• Low Complexity Video Coding and Decoding

61
Distributed Compressed Video Sensing

• The Encoder
• Intra-code Key Frames periodically using
  conventional video compression standards
  (MPEG/H.26x)
• Acquire local block-based and global frame-based
  measurements of CS Frames

Key Frames
MPEG/H.26x IntraCoding
Block-based measurement Acquisition
Transmit to the decoder
Input Video
CS Frames
Frame-based measurement Acquisition
62
Distributed Compressed Video Sensing

• The Decoder
• Decode key frames using conventional image/video
  compression standards
• Perform Sparsity-constraint Block Prediction for
  motion estimation and compensation
• Perform Sparse Recovery with Decoder SI for
  prediction error reconstruction
• Add reconstructed prediction error to the
  block-based prediction frame for final frame
  reconstruction

Block Prediction using the Interframe Sparsity
Model
Key Frames
MPEG/H.26x Decoding
Optimal Block-based Prediction
Output Video
Local block measurements
Sparse recovery with Side Info at the Decoder
Side Info
Measurement Generation
Measurement Union
Global frame measurements
Sparse Recovery of Prediction Error
Measurement Subtraction
63
Distributed Compressed Video Sensing
Encoder
Decoder
Block Prediction using the Interframe Sparsity
Model
Key-frames
Key-frames
MPEG/H.26x IntraCoding
MPEG/H.26x Decoding
Input Video
Output Video
Optimal Block-based Prediction
Block-based measurement Acquisition
Side Info
CS-frames
Measurement Generation
Frame-based measurement Acquisition
Sparse Recovery of Prediction Error
Measurement Subtraction
Measurements Union
Analog-domain Compressive Sensing
Sparse recovery with Side Info at the Decoder
64
Inter-frame Sparsity Model
?B
DB
xB
I-frame
CS-frame
I-frame

• A block XB can be sparsely represented by a
  linear combination of a few temporal neighboring
  blocks
• A generalized model of block motion

XB DB?B
65
Inter-frame Sparsity Model
?B
DB
xB
I-frame
CS-frame
I-frame

b1
b2
b3
b4

• Half-pixel motion compensation

66
Sparsity-Constraint Block Prediction

• Find the block that has the sparsest
  representation in a dictionary of temporal
  neighboring blocks

Received local block measurements
?B Argmin ?B1 s.t. yB FB DB ?B
XB DB?B
Dictionary of temporal neighboring blocks
Block Prediction
Block Sensing Matrix

• A generalized prediction algorithm of both
  full-pixel and sub-pixel best matching block
  search

67
Sparse Recovery with Decoder SI
Frame Prediction (SI)
Measurement Generation
Received Measurements
Sparse Recovery
Prediction Error

• Prediction Error
• Often very sparse
• Can be recovered with higher accuracy

68
Simulation Results
Baseline 27.9 dB

• Performance Comparison
• DISCOS and
• CS-based intra-coding and intra-decoding
  (Baseline)

DISCOS 38.7dB
Reconstruction of frame 41 from 25 measurements
Baseline 27.9 dB DISCOS
38.7dB Fig. 4
Reconstruction of frame 41 from 25 measurements
69
Simulation Results
Baseline 24.3 dB

• Performance Comparison
• DISCOS and
• CS-based intra-coding and intra-decoding
  (Baseline)

DISCOS 32.9dB
Reconstruction of frame 21 from 25 measurements
Baseline 27.9 dB DISCOS
38.7dB Fig. 4
Reconstruction of frame 41 from 25 measurements
70
Error-Resilient Data Transmission
Packet Loss Channel
Enhancement Layer Encoder
Enhancement Layer Decoder

• Compressive Sensing Approach

71
Previous Approaches

• FEC
• Employ well-known channel codes (Reed-Solomon,
  Turbo code or LDPC,etc)
• Decoded video quality significantly degraded when
  packet loss rate higher than error correction
  capacity of the channel code (cliff effect)

All based on coding technique on a finite field

• Recent Approaches
• Wyner-Ziv coding technique based SLEP
  (Stanford), Layered Wyner-Ziv Video Coding (Texas
  AM)
• Distributed Video Coding to mitigate error
  propagation of the predictive video coding PRISM
  (Berkeley)

72
Compressive Sensing Approach
A new channel-coding technique on a REAL FIELD

• Borrow principles from Compressive Sensing
• Effectively mitigate the cliff effect thanks to
  the soft-decoding feature of Sparse Recovery
  Algorithm
• Eliminate Get all or nothing feature of coding
  techniques on a finite field

73
Layered Compressive Sensing Video Codec
Packet Loss Channel
MPEG/H.26x Encoding
MPEG/H.26x Decoding
E-1
Side info
Measurement Generation
Motion vectors, mode decisions
Quantized transformed prediction error
E
E-1
Sparse Recovery
Measurement Acquisition
R
Rounding
Entropy Coding
Motion vectors, mode decisions
Sparse Recovery with Decoder Side Info
74
Base Layer Coding
Slice of MacroBlocks

• Conventionally encoded by video compression
  standards MPEG/H.26x
• Slices of an prediction error frame are
  entropy-coded and packetized before being
  transmitted over error-prone channels without any
  error correcting code

75
Enhancement Layer Coding
A slice of MacroBlocks
Cross-slice measurements

• Measurements
• Acquired across slices of an error prediction
  frame
• Rounded to integers, entropy-coded and sent to
  the decoder (along with motion vectors and mode
  decisions)

76
LACOS Decoder
MPEG/H.26x Decoding
E-1
Side info
Motion vectors, mode decisions
Measurement Generation
E-1
Sparse Recovery

• Entropy-decode a corrupted base layer (regarded
  as SI)
• Feed the SI and cross-slice measurements received
  from the enhancement layer into a sparse recovery
  with decoder SI for recovering lost
  slices/packets
• Add recovered slices/packets back to the
  corrupted base layer for a final reconstruction
  of prediction error frames
• Feed reconstruction of prediction error frames
  into a regular MPEG/H.26x decoder for final video
  frame reconstruction

77
Example of Coding Decoding
Observed
Observed
v argmin v1 s.t. u Fv
x xSI v
Observed
78
Sparse Recovery Algorithm for LACOS

• Sparsity Adaptive Matching Pursuit Algorithm
  (SAMP) (D. Tran)
• Follow the divide and conquer principle through
  stage by stage estimation the sparsity K and the
  true support set
• At each stage, a fixed size finalist of
  nonzero, significant coefficients is iteratively
  refined via the Final Test.
• When energy of a current residue is greater than
  that of previous iteration residue , shift to a
  new stage and expand the size of finalist by a
  step-size s
• Optimal performance guarantee without prior info
  of sparsity K

79
Simulation Results
FEC 29 dB
Performance comparison LACOS, FEC and Error
Concealment with Football sequence. Base layers
is encoded at 2.97 Mbps
LACOS 30.7dB
Reconstruction of the frame 27 with 13.3 packet
loss
80
Simulation Results
FEC 31.3 dB
Performance Comparison LACOS, FEC and Error
Concealment with CIF sequence Mobile. Base
layers is encoded at 3.79 Mbps
LACOS 33 dB
Reconstruction of the frame 34 with 13.9 packet
loss
81
Some Remarks

• WZ-based Approaches (e.g. FEC)
• Work perfectly when packet loss rate is lower
  than error correction capacity of the channel
  code
• Perform error concealment when channel decoder
  fails that results in low performance (cliff
  effect)
• LACOS
• Holds soft-decoding feature of sparse recovery
  algorithm
• Mitigates the cliff effect effectively or the
  decoded video quality gradually degrades when the
  amount of packet loss increases
• Efficient sensing and fast recovery, enabling it
  to work well in real-time scenarios

82
Conclusion

• Compressed sensing
• A different paradigm for data acquisition
• Sample less and compute more
• Simple encoding most computation at decoder
• Exploit a priori signal sparsity
• Universality, robustness
• Compressed sensing applications for multimedia
• One-pixel Camera
• 2D separable measurement ensemble for image/video
• Face/speech recognition
• MRI applications
• Distributed compressed video sensing
• Layered compressed-sensing robust video
  transmission

83
References
In true compressed sensing fashion

• http//www.dsp.ece.rice.edu/cs/