Model Combination in NeuralBased Forecasting

1
Model Combination inNeural-Based Forecasting
ESI XXII (EURO Summer Institute), July 9-25,
2004, Ankara, Turkey
Paulo S.A. FreitasUniversity of Madeira,
Portugalpaulo_at_uma.pt Antonio J.L.
RodriguesUniv. of Lisbon, Portugal
2
Main topics

• Combination of estimates
• Linear combination
• Extended formulation
• Composition of models
• Low-frequency effects
• High-frequency effects nonlinear
  autocorrelations
• Combining decisionsvs. combining predictions
• What (when) to combine?
• Gaussian radial basis function (RBF) networks

3
Examples of real time series
log - Beveridge wheal price index
log - Index of stock prices
log - Money stock
Maximum daily electrical load
4
Temporal data mining

• Large data sets ? Efficiency problems
• Modelling requirements
• Flexible and robust but not too complex
• Efficacy
• Computational efficiency
• In this study
• Linear parametric models
• Recursive/adaptive estimation
• (time-varying parameters)

5
Gaussian radial basis function network
Gaussian RBF
Output sapce
Input space
(model size)
Identification
(hyperparameters)
6
Identification and estimation

• Identification

• Model size
• optimized offline
• Hyperparameters centres and widths

• Heuristic methods (e.g. k-means clustering
  k-NN ...)

• Estimation (supervised learning)

• (Linear) parameters

E.g. Recursive least-squares (RLS),
Covariance addition method (RLS-CA),
Kalman filter (KF), etc.
7
Model combination

• 1. Combination of estimates
• Composition of models
• Combining decisions vs. combining predictions

8
1. Combination of estimates
Paradigms

• Classical approach
• To identify the best model wrt an optimal
  criteria
• Exhaustive optimization
• Possible usefull information is discarded
• Model mixing
• To combine the estimates of two or more
  (sub-optimal) individual models
• Potential gains both in efficiency and accuracy
• Aggregation of diverse information

9
A simple illustration

• Ideally
• White noise
• Sufficiently non-correlated

Convex linear combination
10
Linear combination

• Given M sequences of estimates
• Least squares solution (or through Moore-Penrose
  pseudo-inverse)
• Direct estimation

Particular cases
11
Extended formulation

• Correlated errors

unrestricted
Recursive estimation

• Optimal solutions

12
Model combination

• 1. Combination of estimates
• Composition of models
• Combining decisions
• vs. combining predictions

13
2. Composition of models (model synthesis)

• Pre-processing methods (to induce stationarity)
• Classical approaches
• Deterministic detrending (e.g. linear
  regression)
• Data differencing (e.g. first differencing)
• Alternative approaches
• Pattern differencing ( standardization)
• Stochastic detrending (or pre-filtering)

14
Pre-filtering methodology

• Sequential (two-step) estimation
• Low-frequency effects (dynamic stochastic
  model?Kalman Filter)
• (e.g. DTR)
• High-frequency effects nonlinear
  autocorrelations
• (e.g. RBF?RLS-CA)

• Simultaneous estimation (complete model?KF)

15
An illustration example
Periodogram
Freq.
16
Comparing approaches
RBF Gaussian RBF Network RLS Recursive Least
Squares
17
RMSE(1) Results
18
Model combination

• 1. Combination of estimates
• Composition of models
• Combining decisions
• vs. combining predictions

19
3. Combining decisions vs. combining predictions

• Optimal forecasting ? optimal decision-making
• Cost functions
• Predictive model based on LS or EWLS criteria
• Prescriptive model usu. based on other criteria

LS
D
20
Deriving optimal quantiles

• Optimal decision

21
What (when) to combine?

• Two approaches, given forecasts from several
  models
• combine forecasts derive quantile
• derive quantiles combine them

22
An example

• Simulated time series
• DTR
• with chosen ( ? , NVR)
• Suboptimal models
• DTR with and NVR optimized
• DTR with and NVR optimized

23
Results

• combine forecasts derive quantile ? D 2.18
• derive quantiles combine them ? D 2.20

24
Conclusions

• Concerns accuracy and efficiency
• Models with linear time-varying parameters
• Recursive estimation (and adaptive
  identification)
• Combination of neural predictive models
• Model mixing extension of classical framework
• Model synthesis coupling with dynamic trend
  models
• Optimal decision-making
• Prediction only as a means to support
  decision-making
• The use of more realistic cost functions
• Combining predictions might be preferable to
  combining decisions