Ordinary Least-Squares

1
Ordinary Least-Squares
2
Outline

• Linear regression
• Geometry of least-squares
• Discussion of the Gauss-Markov theorem

3
One-dimensional regression
4
One-dimensional regression
Find a line that represent the best linear
relationship
5
One-dimensional regression

• Problem the data does not go through a line

6
One-dimensional regression

• Problem the data does not go through a line
• Find the line that minimizes the sum

7
One-dimensional regression

• Problem the data does not go through a line
• Find the line that minimizes the sum
• We are looking for that minimizes

8
Matrix notation

• Using the following notations
• and

9
Matrix notation

• Using the following notations
• and
• we can rewrite the error function using linear
  algebra as

10
Matrix notation

• Using the following notations
• and
• we can rewrite the error function using linear
  algebra as

11
Multidimentional linear regression

• Using a model with m parameters

12
Multidimentional linear regression

• Using a model with m parameters

13
Multidimentional linear regression

• Using a model with m parameters

14
Multidimentional linear regression

• Using a model with m parameters
• and n measurements

15
Multidimentional linear regression

• Using a model with m parameters
• and n measurements

16
(No Transcript)
17
(No Transcript)
18
parameter 1
19
parameter 1
measurement n
20
Minimizing
21
Minimizing
22
Minimizing
is flat at
23
Minimizing
is flat at
24
Minimizing
is flat at
does not go down around
25
Minimizing
is flat at
does not go down around
26
Positive semi-definite
In 1-D
In 2-D
27
Minimizing
28
Minimizing
29
Minimizing
30
Minimizing
Always true
31
Minimizing
The normal equation
Always true
32
Geometric interpretation
33
Geometric interpretation

• b is a vector in Rn

34
Geometric interpretation

• b is a vector in Rn
• The columns of A define a vector space range(A)

35
Geometric interpretation

• b is a vector in Rn
• The columns of A define a vector space range(A)
• Ax is an arbitrary vector in range(A)

36
Geometric interpretation

• b is a vector in Rn
• The columns of A define a vector space range(A)
• Ax is an arbitrary vector in range(A)

37
Geometric interpretation

• is the orthogonal projection of b onto
  range(A)

38
The normal equation
39
The normal equation

• Existence has always a
  solution

40
The normal equation

• Existence has always a
  solution
• Uniqueness the solution is unique if the columns
  of A are linearly independent

41
The normal equation

• Existence has always a
  solution
• Uniqueness the solution is unique if the columns
  of A are linearly independent

42
Under-constrained problem
43
Under-constrained problem
44
Under-constrained problem
45
Under-constrained problem

• Poorly selected data
• One or more of the
• parameters are redundant

46
Under-constrained problem

• Poorly selected data
• One or more of the
• parameters are redundant
• Add constraints

47
How good is the least-squares criteria?

• Optimality the Gauss-Markov theorem

48
How good is the least-squares criteria?

• Optimality the Gauss-Markov theorem
• Let and be two sets of random variables
  
• and define

49
How good is the least-squares criteria?

• Optimality the Gauss-Markov theorem
• Let and be two sets of random variables
  
• and define
• If

50
How good is the least-squares criteria?

• Optimality the Gauss-Markov theorem
• Let and be two sets of random variables
  
• and define
• If
• Then is the
• best unbiased linear estimator

51
b
ei
a
no errors in ai
52
b
b
ei
ei
a
a
errors in ai
no errors in ai
53
b
a
homogeneous errors
54
b
b
a
a
homogeneous errors
non-homogeneous errors
55
b
a
no outliers
56
outliers
b
b
a
a
outliers
no outliers