Optimization Multidimensional

1
Optimization (Multidimensional)
http//www.math.iastate.edu/wu/math597.html
http//www.math.iastate.edu/wu/Math597HW0000/index
.htm

• Math/BCB/ComS597
• Zhijun Wu
• Department of Mathematics

2
Multivariable Functions
gradient
real-valued function
vector-valued function
3
g (x)
H (x)
J (x)
Gradient, Hessian, Jacobian
4
Local Optimization
A necessary condition for a point x to be a
minimizer of f is the gradient of f at x is
equal to zero.
x is a local minimizer of f if for any x in a
small neighborhood of x, f (x) gt f (x).

5
Newtons Method
6
Local Convergence
Under some appropriate assumptions, the sequence
of the iterates generated by the Newtons method
for a function f can always converge to a local
minimizer x of f given initial solution x0
sufficiently close to x.
The Newtons method converges locally
quadratically to a local minimizer in the sense
that when it gets sufficiently close to the
minimizer, the distance between the current
iterate and the minimizer reduces quadratically
as the iteration proceeds.
7
Assume H (x) is symmetric positive definite,
i.e., for any vector p ? 0, pTH (x)p gt 0.
Hessian Matrix
H is symmetric positive definite if and only if f
is a strictly convex function.
f is a strictly convex function if for any two
points x and y, f ((1- ?)x?y) lt (1- ?)f (x)
?f (y), 0lt?lt1.
If H is symmetric positive definite, H LLT,
where L is a lower triangular matrix, and H LLT
is called a Cholesky factorization of H.
8
Newton Step
n3 / 3
n2
n2
Matlab
9
newton.m function x, f newton (fun, grad,
hes, x0) x x0 f feval (fun, x) g feval
(grad, x) h feval (hes, x) while norm (g) gt
0.00001 x x - h \ g f feval (fun,
x) g feval (grad, x) h feval (hes,
x) end
10
Wood Function
11
Wood Function
12
wdfun.m function f wdfun (x) f 100 (x (1)
x (1) - x (2)) 2 (1 - x (1)) 2 f f
90 (x (3) x (3) - x (4)) 2 (1 - x (3))
2 f f 10.1 ((1 - x (2)) 2 (1 - x (4))
2) f f 19.8 (1 - x (2)) (1 - x (4))
wdgrad.m function g wdgrad (x) g (1,1) 400
(x (1) x (1) - x (2)) x (1) 2 (1 - x
(1)) g (2,1) - 200 (x (1) x (1) - x (2)) -
20.2 (1 - x (2)) - 19.8 (1 - x (4)) g (3,1)
360 (x (3) x (3) - x (4)) x (3) - 2 (1
- x (3)) g (4,1) - 180 (x (3) x (3) - x
(4)) - 20.2 (1 - x (4)) - 19.8 (1 - x (2))
13
wdhes.m function h wdhes (x) h (1,1) 1200
x (1) x (1) - 400 x (2) 2 h (1,2) - 400
x (1) h (1,3) 0 h (1,4) 0 h (2,1) -
400 x (1) h (2,2) 200 20.2 h (2,3) 0 h
(2,4) 19.8 h (3,1) 0 h (3,2) 0 h (3,3)
1080 x (3) x(3) - 360 x (4) 2 h (3,4)
- 360 x (3) h (4,1) 0 h (4,2) 19.8 h
(4,3) - 360 x (3) h (4,4) 180 20.2
14
gtgt x0 1.5 1.5 1.5 1.5 gtgt x, f newton
(wdfun, wdgrad, wdhes, x0) x
1.0000 1.0000 1.0000 1.0000 f
0 gtgt x0 -1.5 -1.5 -1.5 -1.5 gtgt x, f
newton (wdfun, wdgrad, wdhes, x0) x
-0.9680 0.9471 -0.9695 0.9512 f
7.8770