The WidrowHoff Algorithm Primal Form

1
The Widrow-Hoff Algorithm (Primal Form)
Initial
Repeat
(Typo on textbook!)
Until convergence criterion satisfied return

• Minimize the square loss function using gradient
  descent

)

• Dual form exists (i.e.

2
Gradient and Hessian
3
Example1
4
Example 2
The Hessian is positive semi-definite
5
Solution of the Least Squares Problem
The Normal Equations
Notation
Find such that has the smallest
value, i.e.
This is a quadratic unconstrained minimization
problem is the optimal
solution if and only if
6
The Normal Equations of LSQ
Letting
we have
the normal equations of LSQ
is inversable then
If
Note The above result is based on the First Order
Optimality Conditions (necessary sufficient for
differentiable convex minimization problems)
is singular ?
What if
7
Ridge Regression(Guarantee
Exist)
where