Online learning system for autoregressive model

1
Online learning system for auto-regressive model
Oct. 22, 2002
Chaos and Non-Linear Time Series and Related
Topics
Kaoru Fueda Okayama University
2
Abstract

• For analysis of non-linear auto-regressive time
  series model, Fueda and Yanagawa(2001) gives the
  consistent estimator of the order and other
  parameter of auto-regressive model. Today we make
  improvements for their method.

3
Abstract

• To estimate the non-linear auto-regressive
  function, we need a large number of sample from
  long term observation. In real data, the
  auto-regressive function might change slightly in
  a long term, even if it seems to be stationary
  for a short term. Our method adapts such slight
  change by on-line learning system.

4
Non-linear auto regressive model

• We consider the stochastic mode given by

specially whose deterministic skeleton
makes chaotic model.
5
What is chaos

• Chaos is a complex system made by iteration of a
  simple function.

• Locally it expands small differences, but
  globally bounded.

6
The aim of research

• To estimate the simple function to be iterated
  instead of the complex function made by
  iteration.
• To estimate derivatives of the simple function to
  check that it expands small differences.

7
What is dynamic noise

• It may change the orbit Xt completely due to
  expansion of a small difference.
• Cf. observation noise doesnt change the orbit.

8
Example of chaos

• Henon map

9
Chaotic time series with dynamic noise
10
Chaotic time series with dynamic noise
11
Previous researches

• For non-linear auto-regressive time series model

• Cheng and Tong(1994) has proposed estimators of
  F and d using Nadaraya-Watson kernel estimator
  and cross validation method.
• For non-linear auto-regressive time series model
  with delay time

Fueda and Yanagawa(2001) has proposed estimators
of F,d and t, and proved their consistency.
12
Fueda and Yanagawa(2001)
13
Fueda and Yanagawa(2001)
14
Fueda and Yanagawa(2001)
15
Yonemoto and Yanagawa(2001)

• Yonemoto and Yanagawa(2001) pointed out that
  Fueda and Yanagawa method often fails to estimate
  the true d0 and t0.
• They proposed a new criterion, and confirmed
  their criterion works well by simulation.

16
Yonemoto and Yanagawa(2001)
17
Conditions on the bandwidth h

• Fueda and Yanagawa(2001) needs some conditions on
  bandwidth h to get consistency of estimator. For
  example,

• Yonemoto and Yanagawa(2001) choose h to minimize
  CV(t,t) for each t,t .

18
Multivariate local linear regression

• Since the kernel estimator has bias at the bounds
  of data point, Fueda(2001) has proposed a
  estimator of F,d and t based on the local linear
  regression and bias correction.

19
Multivariate local linear regression
20
Multivariate local linear regression

• We may write D(ß, z, X) as a matrix form

21
Multivariate local linear regression

• D(ß, z, X) is minimized by

and estimators of F and its derivatives are given
by
22
Model selection

• For each d and t, put

23
Multivariate local linear regression

• By this method, we can estimate F(z).
• But often fail to estimate its derivatives.

24
Difficulty in this estimation

• Henon map
• zF(y,x)

Available data points(x,y)
25
Difficulty in this estimation

• Available data points

26
Dimension of data points

• Dimension of data point of chaotic time series is
  a non-ingegre Fractal dimension.
• Definition of Fractal dimension(Grassberger and
  Procaccia(1983))

27
Locally ill condition

• Then we introduce PCA to local linear regression,
  and select variables adaptively.

28
Principal Component Analysis

• For Locally weighted design matrix

29
Locally weighted PCA

• For such matrix V, we may rewrite

30
Adaptive variable selection
31
Adaptive variable selection

• Then D(ß, z, X) is minimized by

32
Adaptive variable selection
33
Adaptive autoregressive model

• To estimate the non-linear auto-regressive
  function, we need a large number of sample form
  long term observation. In real data, the
  auto-regressive function F might slightly change
  in a long term, even if it seems to be stationary
  for a short term. Thus we introduce the on-line
  learning (or forgetting) system to adapt such
  data.

34
Adaptive autoregressive model

• For non-parametric case