Neural Networks as a New Approach for Data Assimilation

1
Neural Networks as a New Approach for Data
Assimilation
(possible)
Haroldo F. de Campos Velho Laboratory for
Computing and Applied Mathematics (LAC) National
Institute for Space Research (INPE)
Brazil E-mail haroldo_at_lac.inpe.br Web-page
http//www.lac.inpe.br/haroldo/
First LNCC Workshop on Computational Modelling
Petrópolis (RJ) 2004
2
Team

• Alexandre G. Nowosad (CPTEC-INPE)
• Fabrício P. Harter (PhD student, LAC-INPE)
• Haroldo F. de Campos Velho (LAC-INPE)
• Rosangela Cintra (CPTEC-INPE)

3
Thematic Scientific Project (Fapesp/BC) -
Applications

• Cooperation project LAC-INPE, COGS Univ.
  Sussex, UNIVAP
• Space Science Determination of the position of
  collect platform of satellite data
• Representation of
  background cosmic radiation in microwave
• Loop Reconstruction
  in solar plasma
• Geophysics Magnetotelluric inversion
• Space Technology Identification of thermal
  properties
• Optimal design
  in satellite thermal analysis
• Damage
  identification in aerospace structures
• Material Science Fault detection in composite
  materials
• Dentritic cristalization Identification of
  diffusivity coefficients
• Oceanography Estimation of optical properties
  in natural waters
• Meteorology Estimation of temperature and
  humidity vertical profiles

4
Thematic Scietific Project (Fapesp/BC) - Methods

• Cooperation project LAC-INPE, COGS Univ.
  Sussex, UNIVAP
• Regularization operator Higher order Tikhonov
  regularization
• 
  Entropy of higher order
• 
  Non-extensive entropy
• Optimizers Deterministic Quasi-Newtonian,
  conjugate gradient (CG),
• 
  Levenberg-Marquadt, Simplex
• Stochastic Simulated
  annealing (SA), genetic algorithms (GA),
• 
  generalized extreme optimization (GEO),
• 
  social insect systems (ant colony system)
• Hybrid SA Simplex
  SAPlex GA Simplex GAPlex
• CG
  GA-epidemic
• Artificial neural networks

5
Scientific challenges
1. Before the XX century We want to know
the nature laws (mechanics, thermodynamics,
electromagnetism, life evolution, social
behaviour, transfinite numbers)
2. During the XX century We know the laws
(equations), but we want to solve them.
Remarkable conquest modern numerical weather
prediction!
3. After the XX century (our century!)
Starting this new century, data analysis is
occuping a central role in the science (genomic,
data mining, background cosmic radiation in
microwave, data assimilation).
6
Data Assimilation (explanation/motivation)
Applications Air monitoring (P. Zannetti Air
Pollution Modeling, 1990) Meteorology (R. Daley
Atmospheric Data Analysis, 1991)
(E. Kalnay Atmospheric Modeling, Data
ssimilation and
Predictability, 2002) Oceanography (A.F. Bennet
Inverse Methods in Physical
Oceanography, 1992)
7
Data assimilation in geophysical fluid dynamics
multi-step process

• Collecting data (observational system ground
  stations, radiosonders, sattelite, ships, ...)
• Objective analysis distributing the
  observations on the computational grid points
• Data assimilation combining two sources of data
  (mathematical model observations analysis)
• Initialization (filtering out high frequencies)

8
Methods for data assimilation

• Newtonian relaxation (nudging)
• Statistical (optimal) interpolation
• Kalman filter
• Variational method 3D and 4D
• New methods for data assimilation
• - Ensemble Kalman filter
• - Artificial neural networks

9
Newtonian relaxation (nudging)
N relaxation coefficient, computed from
numerical experimentation The procedure have
been abandoned, because it is unable to follow a
chaotic dynamics. The idea is to integrate the
model up to a future time (observations),
considering the forcing (last rhs term)
V. Innocentini, E.S. Caetano Neto, F.P. Harter,
Bras. J. Meteorol., 17(2),125-140, 2002. T.N.
Krishnamurti, H.S. Bedi, V.M. Hardiker An
Introduction to Global Spectral Modeling, 1998.
10
Statistical (optimal) interpolation
Data assimilation is a two-step
procedure where dn1 is the
inovation being the observed state,
and Hn represents the observation system
11
Statistical (optimal) interpolation
Discrepancy function The cost function J(x)
for variational methods B background error
covariance matrix R observational error
covariance matrix
12
Kalman filter
Three versions Linear, Extended, Adaptive
A.G. Nowosad, A. Rios Neto, H.F. de Campos Velho
Data Assimilation Using an Adaptative Kalman
Filter and Laplace Transform, Hybrid Methods in
Engineering, 2(3), 291-310, 2000.
13
Variational methods 3D and 4D
Objective (cost) function (adjoint equation will
not discussed here)
Na1 for 3D-Var, and Na gt1 for 4D-Var
. Covariance matrices they can be estimated by
Fokker-Planck equation (K Belyaev, CAS Tanajura
(2002) Appl. Math Model., 26(11)1019-1027) B
background error covariance matrix R
observational error covariance matrix
14
Remarks on OI, 3D-Var, Kalman filter and 4D-Var
Remark-1 OI and 3D-Var could be equivalent,
dealing with optimal least square gain.
Remark-2 extended KF and 4D-Var over a given
time interval, perfect model, the 4D-Var analysis
at the end of the time interval is equal to the
Kalman filter analysis at the same time.
Details http//www.ecmwf.int/newsevents/training/
rcourse_notes/DATA_ASSIMILATION/ASSIM_CONCEPTS/ind
ex.html
15
New methods ensemble Kalman filter
Many scientists believe that EKF will be the
assimilation method for most of operational
centers for numerical weather prediction.
Random perturbations are added to the
observations assimilated. This will allow to
compute the covariance from the ensemble
Nk is of O(10) or O(100), the computational
cost is increased by this order (compared to OI
or 3D-Var. But this larger cost is small compared
to the extended Kalman filter.
16
New methods artificial neural network
For artificial neural networks (ANN), the
analysis step is done by a trained ANN
multi-layer perceptron, backpropagation
propagation algorithm for learning phase
emulating an extended Kalman filter
Training phase determination of the connection
weights, bias Activation phase generating
analized data.
17
Artificial neural network
Activation
Learning
18
Artificial neural network
Testing model Lorenz system
w0 ? X0 Y0 Z0T 1.508870 -1.5312
25.46091T
Euler predictor-corrector method adopting the
following dimensionless quantities ?t0.001,
?10, b8/3, R28, producing a chaotic dynamics.
A.G. Nowosad, A. Rios Neto, H.F. de Campos Velho
(2000) Data Assimilation in Chaotic Dynamics
Using Neural Networks, III ICONE, Brasil, pp.
212-221.
19
Artificial neural network
Testing model shallow water equations

• u, v zonal and meridian wind components
• ? the geopotential ? ?u/?x divergence ?
  ?v/?x vorticity
• Ro0.10 Rossby number RF0.16 Froude number
• R?10 a number associated to the ?-effect
• Numerical parameters ?t100 s and Nx?x L
  10000 Km, Nx32
• Discretization forward and central finite
  difference for time and space.

20
NNs for data assimilation
Lorenz dynamics with 2 different conditions (Y
component) w0 and (w0 ?w)
DYNAMO model temporal corruption at central
point for the u-component, disturbances insertion
at each 4 s.
21
NNs for data assimilation
Numerical results Lorenz system
3 neurons in the hidden layer
22
NNs for data assimilation
Numerical results Shallow water
NN 2 hidden layers 50 neurons for each layer
Numerical experiment was made inserting
observations every 11.1 hours. The
observational data were the same as output data
from the mathematical model added to a Gaussian
deviations with zero mean.
23
NNs for data assimilation
Different learning methods
Standard procedure (p1)
24
NNs for data assimilation
Different learning methods geopotential
Standard procedure (p1)
New procedure for bias (p2)
Error is more significant after some assimilation
cycles!
25
Applications space climate
Sun-Earth interaction
A simple model for plasma instability described
by three-waves coupled.
F.P. Harter, E.L. Rempel, H.F. de Campos Velho,
A. Chian (2004) Data Assimilation in Space
Climate, Geophysical Review Letters submitted.
26
Data assimilation is performed by ANN, emulating
an extended Kalman filter. Three regimes were
investigated periodic (not shown), weak chaos
(not shown), strong chaos
Without assimilation sheme
Assimilation scheme actived
27
Neural networks new results Application to the
shallow water equation 1D
28
Neural networks new results
Without assimilation scheme
With assimilation scheme New
feature the assimilation for ANN is made for
each grid point, reducing the complexity of the
algorithm. Example 3 variables 3 observations
and 3 forecasts, producing 3 assimilated data
for each grid point.
29
Final remarks

• Difficult problem for solving data
  representation!

• Can ANNs be the ultimate solution for data
  assimilation?

• I do not know. But I am convinced that neural
  networks must be investigated as a new method for
  data assimilation.
• ANN interesting features
• - They are intrinsicly parallel
• - ANN can be implemented on hardware device.

30
NNs for data assimilation

• MP -NN and RBF-NN were effective for data
  assimilation.
• Lorenz model convergence after 3 neurons or
  more. Results were improved when using 10
  neurons. Shallow Water model convergence was
  possible only after using 50 neurons in the
  intermediate layers.
• The new learning scheme became the error more
  stable, and it reduced the time for the training
  phase.
• A parallel version for learning phase was
  implemented.
• New architectures Hopfield NN.

31
Acknowledgements