Title: Total Least Squares and Errors-in-Variables Modeling : Problem formulation, Algorithms, and Applications PART II
1Total Least Squares and Errors-in-Variables
Modeling Problem formulation, Algorithms, and
ApplicationsPART II
- By Sabine Van Huffel
- ESAT-SCD(SISTA), K.U.Leuven, Belgium
2Overview
- Total Least Squares Extensions
- Structured TLS
- Case study TLS in Renography
- Conclusions
3Total Least Squares Extensions
- Mixed LS-TLS and extensions submatrix of AB
error-free (Demmel, 87 88) - Generalized TLS AA1A2, A1 error-free,
errors on A2 correlated
(Gallo 81, Fuller 87,
Golub, Hoffman, Stewart 87, -
Van
Huffel Vandewalle, 91) - Weighted (scaled) TLS DA B T, D and T
diagonal, unequal error variances in A and B
(Golub, Van Loan 81, Rao 97, Paige and
Strakos, 01-02) - Restricted TLS error matrix of the form EDEC
(includes equality constraints, LS, TLS, mixed
LS-TLS, ) (Van Huffel, Zha 91) - Total (lp) approximation uses other norms
(Watson, Späth, Osborne, 82) - Nonlinear measurement error model (Caroll et al.
95) AXB?C bilinear TLS approach
min
inconsistent (Fuller 87) adjusted LS
estimator
consistentcorrection for small samples (A.
Kukush, I. Markovsky, Van Huffel, 01) Other
nonlinear models semi-linear, quadratic,
polynomial (S. Zwanzig, A. Kukush, I. Markovsky,
Amari 2002)
4Total Least Squares Extensions (contd.)
- Elementwise-Weighted TLS (differently sized
errors) (M.L.
Rastello, A. Premoli, Kukush, Van Huffel 2002) - Bounded Uncertainties (El Ghaoui 1997, Sayed,
Chandrasekaran, Golub 1997) - Structured TLS (e.g. Toeplitz/Hankel,
displacement rank, regularisation)
- (Rosen, Park, Glick, 96, Lemmerling, De
Moor, N. Mastronardi, Van Huffel, M. Schuermans) - Regularized TLS (truncated TLS, quadratic
eigenvalue problems,)
(Guo, Renaut 2002, P.C. Hansen, D. OLeary, G.
Golub, R. Fierro 1997, D. Sima) - Latency error (equation error) (De Moor,
Lemmerling, A. Yeredor 2002) - Cox proportional Hazards model with EIV (H.
Kuchenhoff 2002) - TLS for large scale problems
(using a preconditioned conjugate gradient
method proposed by A. Björck ,1997) - TLS for large scale Toeplitz systems of equations
(estimate not consistent)
(J. Kamm and J. Nagy, 1998, applied Newton
iterations combined with a bisection scheme and
circulant factorization preconditioners) - (S. Van Huffel and P. Lemmerling, eds, TLS and
EIV modeling, Kluwer 2002)
5Overview
- Total Least Squares Extensions
- Structured TLS
- Case study TLS in Renography
- Conclusions
6Structured TLS
Park, Rosen, Glick 94, Lemmerling 99,
Mastronardi 01)
(Abatzoglou,Mendel 87, De Moor 92,
Structured TLS
- Why structured TLS ?
- is structured and noise on different entries of S
is i.i.d. Gaussian white noise - Example Toeplitz matrix
- Computation constrained nonlinear optimization
(Newton) exploit matrix structure ? displacement
rank - Note STLS solution consistent for affine
structures (Kukush 02)
7Is STLS a simple extension of TLS ?
- fTLS(y) is the objective function we have to
minimize for solving the TLS problem - fSTLS(y) is the objective function we have to
minimize for solving the structured TLS problem
8STLS for structured A, b unstructured
Assume that q lt mn different elements of A are
subject to error, e.g. A Toeplitz q lt mn-1, A
sparse qltltmn represents the
corrections applied to these elements Vector ?
and correction matrix E are equivalent rb-(AE)x
gives rr(?,x) STLS problem
Dqxq diagonal matrix of positive
weights Equivalent to TLS when qmn and p2 In
order to solve STLS, an mxq matrix X is needed
such that ExX? X has the following
characteristics - Elements of X are the xis
with suitable repetition- Number of non-zeros in
X equals number of non-zeros in E- X and E have
similar structure
9Construction of E and X ExX?
If ?k is (i,j)th element of E then xj is (i,k)th
element of X Example
10Solve STLS iteratively by linearizing r(?,x)
- Let ?x, ?E and ?? represent small changs
- Use (?E)xX?? and neglect 2nd order terms in ??,
?x. - Linearization gives
- r(???,x?x) b-(AE?E)(x?x)
- ?r(?,x)-X??-(AE)?x
- At each iteration, solve the linearized
minimization - with ,
rank (M)nq if (AE) is of full rank
11STLS for structured A unstructured b
- Input A, D,b, structure on A, tolerance ?
- Output correction matrix E, solution x
- 1. Set E0, ?0, compute x from X from x, set
rb-Ax - 2. repeat
- (a)
- (b) set (c) construct E from ? and X
from x compute rb-(AE)x until - p2?2(a)LS problem
12Computational efficiency by exploiting structure
of
- step 2.(a) of basic algorithm LS problem
assume p2, A Toeplitz, DI ? exploit low
displacement rank of involved matrices
sparsity of generators ? O(MNN2) flops - comparison in efficiency, using simulation
example- alg1 see above (exploiting
displacement structure sparsity)- alg2
basic algorithm without exploitation of structure
(O(MN)3)
13Overview
- Total Least Squares Extensions
- Structured TLS
- Case study TLS in Renography
- Conclusions
14Case Study TLS in RENOGRAPHY renogram
deconvolution in kidneys
- In collaboration with the division of nuclear
medicine, Univ. Hospital Leuven, Belgium - co-workers P. Lemmerling, N. Mastronardi, J.
Baetens
15Measurement setup
16Measurement setup
17Overview of the renal scintigraphy
18Used renal regions of interest
Heart
Right kidney
Left kidney
Background region
19TAC (Time-Activity) curves
Kidneys y(t)
Heart or renal artery u(t) y(t)u(t)h(t) where
y(t)renal TAC (OUT) u(t)heart TAC
(IN) h(t)impulse response (unknown) convolution
operator
20Impulse response estimation by discrete
deconvolution
- assume system - linear - time invariant -
causal - zero initial state - finite state
dimension
d(t)
1
t
impulse response
impulse
u(t)
21Convolution illustrated
22Example Impulse response estimation
by discrete deconvolution
Measure u(t) and y(t), find h(t)discrete
deconvolution
0
Ymx1
H
Umxn
u(t) and y(t) noisy ? TLS recommended exploit
matrix structure of U ? STLS (max. likelihood)
23Simulation setup
24Comparison in accuracy
MA versus TLS (MAMatrix Algorithm Solves a
square system YUH Via Gaussian Elimination
with Partial pivoting)
- TLS more accurate than MA, even if curves are
smoothed- accuracy of MA depends heavily on the
number of smoothings- TLS needs no smoothing-
overdetermination not possible with MA- MA fails
to solve rank - deficient problems TLS more
reliable
25Comparison in accuracy
Average relative error of MA and TLS in function
of ?v for 4 different degrees of smoothing
26Renogram deconvolution via STLS
- relations between in- and output at time t
y(t)u(t)h(0)u(t-1)h(1)...u(1)h(t-1)u(o)h(t
)
model selection criterion statistically optimal
relation between in- and outputs
structure to preserve
27STLS solution H can be computed via STLS
algorithm provided p2, mM, nN, qM, DI
weighted matrix, ? set to 10-6, AEU?U,
brY?Y, xH
0
0
28TLS versus STLS in renal deconvolution
TLS more reliable robust than currently used
algorithm (Gaussian elim. with partial pivoting)
Add regularisation as noise st. dev. increases
(under study)
29Other STLS Applications
- Medical diagnosis (renography)
- Polysomnography (exponential data modeling)
- System identification (ARMAX modeling)
- Signal Processing (audio, NMR, speech)
- Astronomy
- Information Retrieval
- Image Reconstruction (Deblurring)
- (S. Van Huffel and P. Lemmerling,
eds, TLS and EIV modeling, Kluwer 2002)
30Overview
- Total Least Squares Extensions
- Structured TLS
- Case study TLS in Renography
- Conclusions
31Conclusions Collaboration must continue ...
between STATISTICS, COMPUTATIONAL
MATHEMATICS and ENGINEERING