Automating estimation of warm-up length

1
Automating estimation of warm-up length
Katy Hoad, Stewart Robinson, Ruth Davies Warwick
Business School WSC08
2
Research Aim

• To create an automated system for dealing with
  the problem of initial bias, for implementation
  into simulation software.
• Target audience non- (statistically) expert
  simulation users.

3
The Initial Bias Problem

• Model may not start in a typical state.
• Can cause initial bias in the output.
• Method used Deletion of the initial transient
  data by specifying a warm-up period (or
  truncation point).
• How do you estimate the length of the warm-up
  period required?

4

• Literature search 44 methods
• Short-listing of methods
• Accuracy robustness
• Ease of automation
• Generality
• Computer running time
• Preliminary Testing 6 methods
• MSER-5 most accurate and robust method.

5
(No Transcript)
6
Further Testing of MSER-5

• Artificial data controllable comparable
• initial bias functions
• steady state functions
• Full factorial design.
• Set of performance criteria.

7
1. Artificial Data Parameters
Parameters Levels
Data Type Single run Data averaged over 5 reps
Error type N(1,1), Exp(1)
Auto-correlation None, AR(1), AR(2), MA(2), AR(4), ARMA(5,5)
Bias Severity 1, 2, 4
Bias Length 0, 10, 40, 100 (of n 1000)
Bias direction Positive, Negative
Bias shape 7 shapes
8

• Mean Shift
• Linear
• Quadratic
• Exponential
• Oscillating (decreasing)

9

• Add Initial Bias to Steady state
• Superpostion Bias Fn, a(t), added onto end of
  steady state function
• e.g.
• 2. Full factorial design
• 3048 types of artificial data set
• MSER-5 run with each type 100 times

10
3. Performance Criteria

1. Coverage of true mean.
2. Closeness of estimated truncation point (Lsol) to
   true truncation point (L).
3. Percentage bias removed by truncation.
4. Analysis of the pattern frequency of rejections
   of Lsol (i.e. Lsol gt n/2).

11
MSER-5 Results
i. Coverage of true mean.
Does the true mean fall into the 95 CI for the
estimated mean?
Non-truncated data sets Truncated data sets of cases
yes yes 7.7
no yes 72.5
no no 19.8
yes no 0
12

• ii. Closeness of Lsol to L.
• Wide range of Lsol values.
• e.g.

(Positive bias functions, single run data, N(1,1)
errors, MA(2) auto-correlation, bias severity
value of 2 and true L 100.)
13
iii. Percentage bias removed by truncation.
14

• Effect of data parameters on bias removal
• No significant effect Error type
• Bias direction
• Significant effect Data type
• Auto-correlation type
• Bias shape
• Bias severity
• Bias length

15
More bias removed by using averaged replications
rather than a single run.
16
The stronger the auto-correlation, the less
accurate the bias removal. Effect greatly reduced
by using averaged data.
17
The more sharply the initial bias declines, the
more likely MSER-5 is to underestimate the
warm-up period and to remove increasingly less
bias.
18
As the bias severity increases, MSER-5 removes an
increasingly higher percentage of the bias.
19
Longer bias removed slightly more efficiently
than shorter bias. Shorter bias - more
overestimations - partly due to longer bias
overestimations being more likely to be rejected.
20
iv. Lsol rejections
Rejections caused by high auto-correlation, bias
close to n/2, smooth end to data end point
rejection. Averaged data slightly increases
probability of getting end point rejection but
increases probability of more accurate L
estimates.
21
Giving more data to MSER-5 in an iterative
fashion produces a valid Lsol value where
previously the Lsol value had been rejected. e.g.
ARMA(5,5)
22
Testing MSER-5 with data that has no initial bias.
Want Lsol 0
Lsol values Percentage of cases
Lsol 0 71
Lsol 50 93
Lsol gt 50 mainly due to highest auto-correlated
data sets - AR(1) ARMA(5,5).
Rejected Lsol values 5.6 of the 2400 Lsol
values produced. 93 from the highest
auto-correlated data ARMA(5,5).
23
Testing MSER-5 with data that has 100 bias.
Want 100 rejection rate Actual rate 61
24
Summary

• MSER-5 most promising method for automation
• Not model or data type specific.
• No estimation of parameters needed.
• Can function without user intervention.
• Shown to perform robustly and effectively for the
  majority of data sets tested.
• Quick to run.
• Fairly simple to understand.

25
Heuristic framework around MSER-5
Iterative procedure for procuring more data when
required. Failsafe mechanism - to deal with
possibility of data not in steady state
insufficient data provided when highly
auto-correlated. Being implemented in SIMUL8.
26
ACKNOWLEDGMENTSThis work is part of the
Automating Simulation Output Analysis (AutoSimOA)
project (http//www.wbs.ac.uk/go/autosimoa) that
is funded by the UK Engineering and Physical
Sciences Research Council (EP/D033640/1). The
work is being carried out in collaboration with
SIMUL8 Corporation, who are also providing
sponsorship for the project.
Thank you for listening.

• Katy Hoad, Stewart Robinson, Ruth Davies
• Warwick Business School
• WSC08