OPTIMIZATION SOFTWARE as a Tool for Solving Differential Equations Using NEURAL NETWORKS

1
OPTIMIZATION SOFTWARE as a Tool for Solving
Differential Equations UsingNEURAL NETWORKS
Fotiadis, D. I. Karras, D. A. Lagaris, I. E.
Likas, A. Papageorgiou, D. G.
2
DIFFERENTIAL EQUATIONS HANDLED

• ODEs
• Systems of ODEs
• PDEs ( Boundary and Initial Value Problems
  )
• Eigen - Value PDE Problems
• IDEs

3
ARTIFICIAL NEURAL NETWORKS

• Closed Analytic Form
• Universal Approximators
• Linear and Non-Linear Parameters
• Highly Parallel Systems
• Specialized Hardware for ANN

4
OPTIMIZATION ENVIRONMENT
MERLIN / MCL 3.0 SOFTWARE

• Features Include
• A Host of Optimization Algorithms
• Special Merit for Sums of Squares
• Variable Bounds and Variable Fixing
• Command Driven User Interface
• Numerical Estimation of Derivatives
• Dynamic Programming of Strategies

5
ARTIFICIALNEURAL NETWORKS

• Inspired from biological NN

Input - Output mapping via the weights u,w,v and
the activation functions s
Analytically this is given by the formula
6
Activation Functions
Many different functions can be used. Our current
choice The Sigmoidal
A smooth function, infinitely differentiable,
bounded in (0,1)
7
The Sigmoidal properties
8
FACTS
Kolmogorov and Cybenko and Hornik
proved theorems concerning the approximation
capabilities of ANNs
In fact it is shown that ANNs are UNIVERSAL
APPROXIMATORS
9
DESCRIPTION OF THE METHOD
SOLVE THE EQUATION
SUBJECT TO DIRICHLET B.C.
Where L is an Integrodifferential
Operator Linear or Non-Linear
10

• Where
• B(x) satisfies the BC
• Z(x) vanishes on the boundary
• N(x) is an Artificial Neural Net

11
MODEL PROPERTIES
The Model satisfies by construction the B.C.
The Model thanks to the Network is trainable
The Network parameters can be adjusted so that
12
The residual Error
13
ILLUSTRATION
Simple 1-d example
Model
14
ILLUSTRATION
For a second order, two-dimensional PDE
where
15
EXAMPLES
Problem Solve the 2-d PDE
In the domain
Subject to the BC
A single hidden layer Perceptron was used
16
GRAPHICAL REPRESENTATION
The analytic solution is
Exact
17
GRAPHS COMPARISON
Neural Solution accuracy Plot Points Training
Points
18
GRAPHS COMPARISON
Neural Solution accuracy Plot Points Test Points
19
GRAPHS COMPARISON
Finite Element Solution accuracy Plot Points
Training Points
20
GRAPHS COMPARISON
Finite Element Solution accuracy Plot Points
Test Points
21
PERFORMANCE

• Highly Accurate Solution (even with few
  training points)
• Uniform Error Distribution
• Superior Interpolation Properties

The model solution is very flexible. Can be
easily enhanced to offer even higher accuracy.
22
EIGEN VALUE PROBLEMS
The model is the same as before. However the
Error is defined as
23
EIGEN VALUE PROBLEMS
Where
i.e. the value for which the Error is minimum.
Problems of that kind are often encountered in
Quantum Mechanics. (Schrödingers equation)
24
EXAMPLES
The non-local Schrödinger equation
Describes the bound na system in the framework
of the Resonating Group Method.
Model
Where
is a single hidden layer, sigmoidal Perceptron
25
OBTAINING EIGENVALUES
ExampleThe Henon-Heiles potential
Asymptotic behavior
Model used
Use the above model to obtain an eigen solution
F. Obtain a different eigen solution by
deflation, i.e.
This model is orthogonal to F(x,y) by
construction. The procedure can be applied
repeatedly.
26
ARBITRARILY SHAPED DOMAINS
For domains other than Hypercubes the BC cannot
be embedded in the model.
be the set of points
Let
defining the arbitrarily shaped boundary. The BC
are then
We describe two ways to proceed solving the
problem
27
OPTIMIZATION WITH CONSTRAINTS
Model
Error to be minimized Domain terms
Boundary terms
With b a penalty parameter, to control the
degree of satisfaction of the BC.
28
PERCEPTRON-RBF SYNERGY
Model
Where the as are determined in a way so that
the model satisfies the BC exactly, i.e.
The free parameter l is chosen once initially so
as the system above is easily solved.
Error
29
Pros Cons . . .
The RBF - Synergy is

• Computationally costly. A linear system is
  solved each time the model is evaluated.
• Exact in satisfying the BC.

The Penalty method is

• Approximate in satisfying the BC.
• Computationally efficient

30
IN PRACTICE . . .

• Initially proceed via the penalty method, till
  an approximate solution is found.
• Refine the solution, using the RBF- Synergy
  method, to satisfy the BC exactly.

Conclusions Experiments on several model
problems shows performance similar to the one
reported earlier.
31
GENERALOBSERVATIONS

• Enhanced generalization performance is achieved,
  when the exponential weights of the Neural
  Networks are kept small.
• Hence box-constrained optimization methods
  should be applied.
• Bigger Networks (greater number of nodes) can
  achieve higher accuracy.
• This favors the use of
• Existing Specialized Hardware
• Sophisticated Optimization Software

32
MERLIN 3.0
A software package offering many optimization
algorithms and a friendly user interface.
What is it ?
What problems does it solve ?
Find a local minimum of the function
Under the conditions
33
ALGORITHMS
Direct Methods

• SIMPLEX
• ROLL

Gradient Methods
Conjugate Gradient
Quasi Newton

• BFGS (3 versions)
• DFP

• Polak-Ribiere
• Fletcher-Reeves
• Generalized PR

Levenberg-Marquardt

• For Sum-Of-Squares

34
THE USERS PART
What the user has to do ?

• Program the objective function
• Use Merlin to find an optimum

What the user may want to do ?

• Program the gradient
• Program the Hessian
• Program the Jacobian

35
MERLIN FEATURES TOOLS

• Intuitive free-format I/O
• Menu assisted Input
• On-line HELP
• Several gradient modes
• Confidence parameter intervals
• Box constraints
• Postscript graphs
• Programmability
• Open to user enhancements

36
Merlin Control Language
MCL
What is it ?
High-Level Programming Language, that Drives
Merlin Intelligently.
What are the benefits ?

• Abolishes User Intervention.
• Optimization Strategies.
• Handy Utilities.
• Global Optimum Seeking Methods.

37
MCL REPERTOIRE
MCL command types

• Merlin Commands
• Conditionals (IF-THEN-ELSE-ENDIF)
• Loops (DO type of loops)
• Branching (GO TO type)
• I/O (READ/WRITE)

MCL intrinsic variables
All Merlin important variables, e.g.
Parameters, Value, Gradient, Bounds ...
38
SAMPLE MCL PROGRAM
program var i sml bfgs_calls nfix
max_calls sml 1.e-4 Gradient
threshlod. bfgs_calls 1000 Number of BFGS
calls. max_calls 10000 Max. calls to
spend. again loosall nfix 0 loop i
from 1 to dim if absgradi lt sml then
fix (x.i) nfix nfix1 end
if end loop if nfix dim then
display 'Gradient below threshold...'
loosall finish end if bfgs
(nocbfgs_calls) when pcount lt max_calls just
move to again display 'We probably
failed...' end
39
MERLIN-MCLAvailability
The Merlin - MCL package is written in ANSI
Fortran 77 and can be downloaded from the
following URL
http//nrt.cs.uoi.gr/merlin/
It is maintained, supported and is FREELY
available to the scientific community.
40
FUTURE DEVELOPMENTS

• Optimal Training Point Sets
• Optimal Network Architecture
• Expansion Pruning Techniques

Hardware Implementation on NEUROPROCESSORS