Linear Least Squares QR Factorization

1
Linear Least Squares QR Factorization
2
(No Transcript)
3
Systems of linear equations

• Problem to solve M x b
• Given M x b
• Is there a solution?
• Is the solution unique?

4
Systems of linear equations

• Find a set of weights x so that the weighted sum
  of the
• columns of the matrix M is equal to the right
  hand side b

5
Systems of linear equations - Existence
A solution exists if
There exist weights, x1, ., xN, such that

• A solution exists when b is in the span of the
  columns of M

6
Systems of linear equations - Uniqueness
Suppose there exist weights, y1, ., yN, not all
zero, such that
Then Mx b ? Mx My b ? M(xy) b
A solution is unique only if the columns of M
are linearly independent.
7
QR factorization 1

• A matrix Q is said to be orthogonal if its
  columns are orthonormal, i.e. QTQI.
• Orthogonal transformations preserve the Euclidean
  norm since
• Orthogonal matrices can transform vectors in
  various ways, such as rotation or reflections but
  they do not change the Euclidean length of the
  vector. Hence, they preserve the solution to a
  linear least squares problem.

8
QR factorization 2

• Any matrix A(mn) can be represented as
• A QR
• ,where Q(mn) is orthonormal and R(nn) is upper
  triangular

9
QR factorization 2

• Given A , let its QR decomposition be given as
  AQR, where Q is an (m x n) orthonormal matrix
  and R is upper triangular.
• QR factorization transform the linear least
  square problem into a triangular least squares.
• QRx b
• Rx QTb
• xR-1QTb

Matlab Code
10
Normal Equations

• Consider the system
• It can be a result of some physical measurements,
  which usually incorporate some errors. Since, we
  can not solve it exactly, we would like to
  minimize the error
• rb-Ax
• r2rTr(b-Ax)T(b-Ax)bTb-2xTATbxTATAx
• (r2)x0 - zero derivative is a (necessary)
  minimum condition
• -2ATb2ATAx0
• ATAx ATb Normal Equations

11
Normal Equations 2

• ATAx ATb Normal Equations

12
Least squares via AQR decomposition

• A(m,n)Q(m,n)R(n,n), Q is orthogonal, therefore
  QTQI.
• QRxb
• R(n,n)xQT(n,m)b(m,1) -well defined linear
  system
• xR-1QTb
• Q is found by GramSchmidt orthogonalization of
  A. How to find R?
• QRA
• QTQRQTA, but Q is orthogonal, therefore QTQI
• RQTA
• R is upper triangular, since in orthogonalization
  procedure only
• a1,..ak (without ak1,) are used to produce qk

13
Least squares via AQR decomposition 2

• Let us check the correctness
• QRxb
• RxQTb
• xR-1QTb

14
Last lecture reminderQR Factorization By
picture
15
QR Factorization Minimization ViewMinimization
Algorithm
For i 1 to N For each Target Column
For j 1 to i-1 For each Source Column left
of target end end
Orthogonalize Search Direction
Normalize