Total Least Squares and Errors-in-Variables Modeling : Problem formulation, Algorithms, and Applications PART II

1
Total Least Squares and Errors-in-Variables
Modeling Problem formulation, Algorithms, and
ApplicationsPART II

• By Sabine Van Huffel
• ESAT-SCD(SISTA), K.U.Leuven, Belgium

2
Overview

• Total Least Squares Extensions
• Structured TLS
• Case study TLS in Renography
• Conclusions

3
Total Least Squares Extensions

• Mixed LS-TLS and extensions submatrix of AB
  error-free (Demmel, 87 88)
• Generalized TLS AA1A2, A1 error-free,
  errors on A2 correlated
  (Gallo 81, Fuller 87,
  Golub, Hoffman, Stewart  87,
• 
  Van
  Huffel Vandewalle, 91)
• Weighted (scaled) TLS DA B T, D and T
  diagonal, unequal error variances in A and B
  (Golub, Van Loan  81, Rao  97, Paige and
  Strakos, 01-02)
• Restricted TLS error matrix of the form EDEC
  (includes equality constraints, LS, TLS, mixed
  LS-TLS, ) (Van Huffel, Zha  91)
• Total (lp) approximation uses other norms
  (Watson, Späth, Osborne, 82)
• Nonlinear measurement error model (Caroll et al.
   95) AXB?C bilinear TLS approach
  min
  
  inconsistent (Fuller 87) adjusted LS
  estimator
  consistentcorrection for small samples (A.
  Kukush, I. Markovsky, Van Huffel, 01) Other
  nonlinear models semi-linear, quadratic,
  polynomial (S. Zwanzig, A. Kukush, I. Markovsky,
  Amari 2002)

4
Total Least Squares Extensions (contd.)

• Elementwise-Weighted TLS (differently sized
  errors) (M.L.
  Rastello, A. Premoli, Kukush, Van Huffel 2002)
• Bounded Uncertainties (El Ghaoui 1997, Sayed,
  Chandrasekaran, Golub 1997)
• Structured TLS (e.g. Toeplitz/Hankel,
  displacement rank, regularisation)
  
• (Rosen, Park, Glick, 96, Lemmerling, De
  Moor, N. Mastronardi, Van Huffel, M. Schuermans)
• Regularized TLS (truncated TLS, quadratic
  eigenvalue problems,)
  (Guo, Renaut 2002, P.C. Hansen, D. OLeary, G.
  Golub, R. Fierro 1997, D. Sima)
• Latency error (equation error) (De Moor,
  Lemmerling, A. Yeredor 2002)
• Cox proportional Hazards model with EIV (H.
  Kuchenhoff 2002)
• TLS for large scale problems
  
  (using a preconditioned conjugate gradient
  method proposed by A. Björck ,1997)
• TLS for large scale Toeplitz systems of equations
  (estimate not consistent)
  (J. Kamm and J. Nagy, 1998, applied Newton
  iterations combined with a bisection scheme and
  circulant factorization preconditioners)
• (S. Van Huffel and P. Lemmerling, eds, TLS and
  EIV modeling, Kluwer 2002)

5
Overview

• Total Least Squares Extensions
• Structured TLS
• Case study TLS in Renography
• Conclusions

6
Structured TLS
Park, Rosen, Glick  94, Lemmerling 99,
Mastronardi 01)
(Abatzoglou,Mendel  87, De Moor  92,
Structured TLS

• Why structured TLS ?
• is structured and noise on different entries of S
  is i.i.d. Gaussian white noise
• Example Toeplitz matrix
• Computation constrained nonlinear optimization
  (Newton) exploit matrix structure ? displacement
  rank
• Note STLS solution consistent for affine
  structures (Kukush 02)

7
Is STLS a simple extension of TLS ?

• fTLS(y) is the objective function we have to
  minimize for solving the TLS problem
• fSTLS(y) is the objective function we have to
  minimize for solving the structured TLS problem

8
STLS for structured A, b unstructured
Assume that q lt mn different elements of A are
subject to error, e.g. A Toeplitz q lt mn-1, A
sparse qltltmn represents the
corrections applied to these elements Vector ?
and correction matrix E are equivalent rb-(AE)x
gives rr(?,x) STLS problem
Dqxq diagonal matrix of positive
weights Equivalent to TLS when qmn and p2 In
order to solve STLS, an mxq matrix X is needed
such that ExX? X has the following
characteristics - Elements of X are the xis
with suitable repetition- Number of non-zeros in
X equals number of non-zeros in E- X and E have
similar structure
9
Construction of E and X ExX?
If ?k is (i,j)th element of E then xj is (i,k)th
element of X Example
10
Solve STLS iteratively by linearizing r(?,x)

• Let ?x, ?E and ?? represent small changs
• Use (?E)xX?? and neglect 2nd order terms in ??,
  ?x.
• Linearization gives
• r(???,x?x) b-(AE?E)(x?x)
• ?r(?,x)-X??-(AE)?x
• At each iteration, solve the linearized
  minimization
• with ,
  rank (M)nq if (AE) is of full rank

11
STLS for structured A unstructured b

• Input A, D,b, structure on A, tolerance ?
• Output correction matrix E, solution x
• 1. Set E0, ?0, compute x from X from x, set
  rb-Ax
• 2. repeat
• (a)
• (b) set (c) construct E from ? and X
  from x compute rb-(AE)x until
• p2?2(a)LS problem

12
Computational efficiency by exploiting structure
of

• step 2.(a) of basic algorithm LS problem
  assume p2, A Toeplitz, DI ? exploit low
  displacement rank of involved matrices
  sparsity of generators ? O(MNN2) flops
• comparison in efficiency, using simulation
  example- alg1 see above (exploiting
  displacement structure sparsity)- alg2
  basic algorithm without exploitation of structure
  (O(MN)3)

13
Overview

• Total Least Squares Extensions
• Structured TLS
• Case study TLS in Renography
• Conclusions

14
Case Study TLS in RENOGRAPHY renogram
deconvolution in kidneys

• In collaboration with the division of nuclear
  medicine, Univ. Hospital Leuven, Belgium
• co-workers P. Lemmerling, N. Mastronardi, J.
  Baetens

15
Measurement setup
16
Measurement setup
17
Overview of the renal scintigraphy
18
Used renal regions of interest
Heart
Right kidney
Left kidney
Background region
19
TAC (Time-Activity) curves
Kidneys y(t)
Heart or renal artery u(t) y(t)u(t)h(t) where
y(t)renal TAC (OUT) u(t)heart TAC
(IN) h(t)impulse response (unknown) convolution
operator
20
Impulse response estimation by discrete
deconvolution

• assume system - linear - time invariant -
  causal - zero initial state - finite state
  dimension

d(t)
1
t
impulse response
impulse
u(t)
21
Convolution illustrated
22
Example Impulse response estimation
by discrete deconvolution

Measure u(t) and y(t), find h(t)discrete
deconvolution
0
Ymx1
H
Umxn
u(t) and y(t) noisy ? TLS recommended exploit
matrix structure of U ? STLS (max. likelihood)
23
Simulation setup
24
Comparison in accuracy
MA versus TLS (MAMatrix Algorithm Solves a
square system YUH Via Gaussian Elimination
with Partial pivoting)
- TLS more accurate than MA, even if curves are
smoothed- accuracy of MA depends heavily on the
number of smoothings- TLS needs no smoothing-
overdetermination not possible with MA- MA fails
to solve rank - deficient problems TLS more
reliable
25
Comparison in accuracy
Average relative error of MA and TLS in function
of ?v for 4 different degrees of smoothing

26
Renogram deconvolution via STLS

• relations between in- and output at time t
  y(t)u(t)h(0)u(t-1)h(1)...u(1)h(t-1)u(o)h(t
  )

model selection criterion statistically optimal
relation between in- and outputs
structure to preserve
27
STLS solution H can be computed via STLS
algorithm provided p2, mM, nN, qM, DI
weighted matrix, ? set to 10-6, AEU?U,
brY?Y, xH
0
0
28
TLS versus STLS in renal deconvolution
TLS more reliable robust than currently used
algorithm (Gaussian elim. with partial pivoting)
Add regularisation as noise st. dev. increases
(under study)
29
Other STLS Applications

• Medical diagnosis (renography)
• Polysomnography (exponential data modeling)
• System identification (ARMAX modeling)
• Signal Processing (audio, NMR, speech)
• Astronomy
• Information Retrieval
• Image Reconstruction (Deblurring)
• (S. Van Huffel and P. Lemmerling,
  eds, TLS and EIV modeling, Kluwer 2002)

30
Overview

• Total Least Squares Extensions
• Structured TLS
• Case study TLS in Renography
• Conclusions

31
Conclusions Collaboration must continue ...
between STATISTICS, COMPUTATIONAL
MATHEMATICS and ENGINEERING