Least Squares Estimation

1
Least Squares Estimation

• S-88.4221 Postgraduate Seminar on Signal
  Processing

Alexandra Oborina
This presentation is based on 3rd chapter of the
book Dan Simon, Optimal State Estimation
Kalman, Hinf, and Nonlinear Approaches
2
Content

• Estimation of a constant vector
• Weighted least square estimation
• Recursive least square estimation
• Wiener filtering
• Homework

3
Estimation of a constant

• Given x-constant, unknown vector, y-noisy
  measurement vector. Problem find best estimate
  of x.

4
Example
5
Weighted LS Estimation

• Given x-constant, unknown vector, y-noisy
  measurement vector. The variance of the
  measurement noise may be different for each
  element of y. Problem find best estimate of x.

6
Example
7
Recursive LS Estimation

• What if we get measurements sequentially?
• Suppose is given after k-1 measurements. So
  new yk is obtained. How to update the estimate
  ?

8
Recursive LS Estimation
9
Recursive LS Estimation
10
Recursive LS EstimationAlternative forms

• Using matrix inversion lemma, substitution and
  inversion alternative forms for Kk and Pk can be
  obtained.
• Alternative forms are mathematically identical,
  but can be beneficial from computational point of
  view.

11
Recursive LS EstimationAlgorithm

• Initialize the estimator as
• For k1,2,.. perform
• Obtain the measurements with white noise

12
Recursive LS EstimationAlgorithm

• Update the estimate of x and estimation-error
  covariance

13
Recursive LS Estimation
14
Example 1
15
Example 1

• By induction
• If x is known perfectly a priori

16
Example 1

• If x is completely unknown a priory

17
Example 1
18
Example 2 - Linear data fitting

• Suppose we want to fit a straight line to a set
  of data points
• Problem find linear relation between yk and tk,
  that means estimate the constants x1 and x2

19
Example 2 - Linear data fitting

• Recursive LS initialization
• Using equations (1), (3), (4) perform recursion

20
Example 3 - Quadratic data fitting

• Suppose, a priory is known that the data is a
  quadratic function of time

21
Wiener Filtering

• Problem design a stable LTI filter to extract a
  signal from noise.

22
Wiener Filtering - parametric filter optimization

• Lets find optimal G(w) as a first order, stable,
  causal filter with 1/T BW
• Suppose also the following forms for
• Recall
• Substitute everything to E(e2(t)) and
  differentiate with respect to t

23
Wiener Filtering - general filter optimization

• Problem find filter g(t) that minimize E(e2(t))
• Replace g(t) with

24
Wiener Filtering - noncausal filter optimization

• Noncausal filter means
• So,
• Thus, quantity inside squire brackets must be zero

25
Wiener Filtering - causal filter optimization

• Causal filter means g(t)0 for tlt0. So,
• Let denote some function a(t), that is 0 for tgt0
  and arbitrary for tlt0.

26
Example

• The signal and noise power spectra are given as

27
Homework

• For recursive LS estimation decide is estimator
  unbiased or not.
• For recursive LS estimation show explicitly
  calculations of
• For Wiener filtering prove that