QUASI MAXIMUM LIKELIHOOD

1
QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION
Alexander Bronstein
2
2
BIBLIOGRAPHY
A. Bronstein, M. Bronstein, and M. Zibulevsky,
"Quasi maximum likelihood blind deconvolution
super- ans sub-Gaussianity vs. asymptotic
stability", submitted to IEEE Trans. Sig. Proc.
A. Bronstein, M. Bronstein, M. Zibulevsky, and
Y. Y. Zeevi, "Quasi maximum likelihood blind
deconvolution asymptotic performance analysis",
submitted to IEEE Trans. Information Theory. A.
Bronstein, M. Bronstein, and M. Zibulevsky,
"Relative optimization for blind deconvolution",
submitted to IEEE Trans. Sig. Proc. A.
Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y.
Zeevi, "Quasi maximum likelihood blind
deconvolution of images acquired through
scattering media", Submitted to ISBI04. A.
Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y.
Zeevi, "Quasi maximum likelihood blind
deconvolution of images using optimal sparse
representations", CCIT Report No. 455 (EE No.
1399), Dept. of Electrical Engineering, Technion,
Israel, December 2003. A. Bronstein, M.
Bronstein, and M. Zibulevsky, "Blind
deconvolution with relative Newton method", CCIT
Report No. 444 (EE No. 1385), Dept. of Electrical
Engineering, Technion, Israel, October 2003.
3
3
AGENDA
Introduction
4
4
BLIND DECONVOLUTION PROBLEM
source signal convolution kernel observed
signal sensor noise signal
restoration kernel source estimate arbitrary
scaling factor arbitrary delay
5
5
APPLICATIONS
Acoustics, speech processing ? DEREVERBERATION
Optics, image processing, biomedical imaging ?
DEBLURRING
Communications ? CHANNEL EQUALIZATION
Control ? SYSTEM IDENTIFICATION
Statistics, finances ? ARMA ESTIMATION
6
6
AGENDA
Introduction
QML blind deconvolution
7
7
ML BLIND DECONVOLUTION
ASSUMPTIONS
is i.i.d. with probability density function
has no zeros on the unit circle, i.e.
No noise (precisely no noise model)
is zero-mean
MAXIMUM-LIKELIHOOD BLIND DECONVOLUTION
8
8
QUASI ML BLIND DECONVOLUTION
PROBLEMS OF MAXIMUM LIKELIHOOD
QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION
9
9
THE CHOICE OF ? (s)
10
10
EQUIVARIANCE
QML estimator of given the observation
Theorem The QML estimator is
equivariant, i.e., for every invertible kernel
, it holds where stands for the
impulse response of the inverse of .
ANALYSIS OF ANALYSIS
OF
11
11
GRADIENT HESSIAN OF
GRADIENT
where is the mirror operator.
HESSIAN
12
12
AGENDA
Introduction
QML blind deconvolution
Asymptotic analysis
13
13
ASYMPTOTIC HESSIAN AT THE SOLUTION POINT
At the solution point,
For a sufficiently large sample size , the
Hessian becomes
where
14
14
ASYMPTOTIC ERROR COVARIANCE
Exact restoration kernel
Estimation kernel from the data
The scaling factor has to obey
15
15
ASYMPTOTIC ERROR COVARIANCE
Estimation error
From second-order Taylor expansion,
equivariance
16
16
ASYMPTOTIC ERROR COVARIANCE
Asymptotically ( ),
Separable structure
17
17
ASYMPTOTIC ERROR COVARIANCE
The estimation error covariance matrix
asymptotically separates to
18
18
ASYMPTOTIC ERROR COVARIANCE
Asymptotic gradient covariance matrices
where
19
19
ASYMPTOTIC ERROR COVARIANCE
Asymptotic estimation error covariance
Asymptotic signal-to-interference ratio (SIR)
estimate
20
20
CRAMER-RAO LOWER BOUNDS
True ML estimator
The distribution-dependent parameters simplify to
Asymptotic error covariance simplifies to
where
21
21
CRAMER-RAO LOWER BOUNDS
Asymptotic SIR estimate simplifies to
22
22
SUPEREFFICIENCY
Let the source be sparse, i.e.,
Let be the smoothed absolute value
with smoothing parameter
In the limit
23
23
SUPEREFFICIENCY
Similar results are obtained for
uniformly-distributed source with
Can be extended for sources with PDF vanishing
outside some interval.
24
24
ASYMPTOTIC STABILITY
The QML estimator is said to be
asymptotically stable if is a local
minimizer of in the limit
.
Theorem The QML estimator is
asymptotically stable if the following conditions
hold and is asymptotically unstable if one
of the following conditions hold
25
25
EXAMPLE
Generalized Laplace distribution
26
26
STABILITY OF THE SUPER-GAUSSIAN ESTIMATOR
SUB-GAUSSIAN
SUPER-GAUSSIAN
27
27
STABILITY OF THE SUB-GAUSSIAN ESTIMATOR
SUB-GAUSSIAN
SUPER-GAUSSIAN
28
28
PERFORMANCE OF THE SUPER-GAUSSIAN ESTIMATOR
SUPER-GAUSSIAN
29
29
PERFORMANCE OF THE SUB-GAUSSIAN ESTIMATOR
SUB-GAUSSIAN
30
30
AGENDA
Introduction
QML blind deconvolution
Asymptotic analysis
Relative Newton
31
31
RELATIVE OPTIMIZATION (RO)
Start with and
For until convergence
Start with
Find such that
Update source estimate
End For
Restoration kernel estimate
Source estimate
32
32
RELATIVE OPTIMIZATION (RO)
Observation The k-th step of the relative
optimization algorithm depends only on
Proposition The sequence of target function
values produced by the relative optimization
algorithm is monotonically decreasing, i.e.,
33
33
RELATIVE NEWTON
Relative Newton use one Newton step in the RO
algorithm
Near the solution point
Newton system separates to
34
34
FAST RELATIVE NEWTON
Fast relative Newton use one Newton step with
approximate Hessian in the RO algorithm
regularized approximate Newton system solution.
Approximate Hessian evaluation order of
gradient evaluation
35
35
AGENDA
Introduction
QML blind deconvolution
Asymptotic analysis
Relative Newton
Generalizations
36
36
GENERALIZATIONS
IIR KERNELS
BLOCK PROCESSING ? ONLINE DECONVOLUTION
MULTI-CHANNEL DECONVOLUTION ? BSSBD
DECONVOLUTION OF IMAGES USE OF SPARSE
REPRESENTATIONS
37
GENERALIZATIONS IIR KERNELS
IIR
FIR
38
GENERALIZATIONS ONLINE PROCESSING
Fast Relative Newton (block)
39
GENERALIZATIONS DECONVOLUTION OF IMAGES
SOURCE
OBSERVATION
RESTORATION
40
GENERALIZATIONS DECONVOLUTION OF IMAGES
Fast relative Newton
Fast relative Newton
Newton
Newton