Automating the Analysis of Simulation Output Data

1
Automating the Analysis of Simulation Output Data

• Stewart Robinson, Katy Hoad,
• Ruth Davies
• OR48, September 2006

2
Outline

• The problem
• A prototype automated output Analyser
• Findings from prototype Analyser
• The AutoSimOA Project
• Current work -
• Collecting and characterising real and artificial
  models

3
The Problem

• Prevalence of simulation software
  easy-to-develop models and use by non-experts.

Simulation software generally have very limited
facilities for directing/advising on simulation
experiments.
Main exception is directing scenario selection
through optimisers.
With a lack of the necessary skills and support,
it is highly likely that simulation users are
using their models poorly.
4
The Problem
Despite continued theoretical developments in
simulation output analysis, little is being put
into practical use.

• There are 3 factors that seem to inhibit the
  adoption of output analysis methods
• Limited testing of methods
• Requirement for detailed statistical knowledge
• Methods generally not implemented in simulation
  software (AutoMod/AutoStat is an exception)

A solution would be to provide an automated
output Analyser.
5
A Prototype Analyser

• Masters Project (3 students).
• The Analyser looked at
• Warm-up
• Run-length
• Number of replications
• Scenario analysis could be added.

6
A Prototype Analyser
A prototype Analyser has been developed in
Microsoft Excel. At present it links to the
SIMUL8 software, but it could be used with any
software that can be controlled from Excel VBA.

7
Illustration Warm-up
Load Analyser into Excel. Enter name of SIMUL8
model. Specify initial number of replications and
run-length to use.
8
Illustration Warm-up
9
Illustration Replications
10
Findings from Prototype Analyser
It is possible to link an Automated Analyser in
Excel to a simulation software tool.
This was just a proof of concept.

• Key issues to address
• More thorough testing of output analysis methods
  for their accuracy and their generality.
• Adaptation of methods to sequential procedures
  and to minimise the need for user intervention.

11
The AutoSimOA Project
A 3 year, EPSRC funded project (GR EP/D033640/1)
in collaboration with SIMUL8 Corporation.

• Objectives
• To determine the most appropriate methods for
  automating simulation output analysis
• To determine the effectiveness of the analysis
  methods
• To revise the methods where necessary in order to
  improve their effectiveness and capacity for
  automation
• To propose a procedure for automated output
  analysis of warm-up, replications and run-length
• Only looking at analysis of a single scenario

12
The AutoSimOA Project

• CURRENT WORK
• Literature review of warm-up, replications and
  run-length methods.
• Development of artificial data sets
  (Auto-Regressive Moving average M/M/n/p
  Queues)
• Collection of real simulation models.

13
Use models / data sets

• Provide a representative and sufficient set of
  models / data output for use in discrete event
  simulation research.
• Use models / data sets to test the chosen
  simulation output analysis methods in the
  AutoSimOA Project..

14
Auto Correlation
Spread round mean
In/out of control
Terminating
Group B
Non-terminating
Cycling/Seasonality
Normality
Transient
Steady state
Trend
15
(No Transcript)
16

• ARTIFICIAL MODELS
• Create simple simulation models where
  theoretical value of some attribute is known.
  E.g. M/M/1 mean waiting time.
• Create simple simulation models where value of
  some attribute is estimated but model
  characteristics can be controlled. E.g.
  Single item inventory management system
  Number-in-stock.
• Create data sets from known equations, which
  closely resembles real model output, with known
  value for some specific attribute. E.g. AR(1)
  with Normal errors

Create different output types
Transient
Steady state cycle
Trend
Steady state
Initial transient (warm-up)
17
Example artificial models
1. E4 Erlang(4) / M / 1 Queue
mean 1.8
Traffic Intensity 0.8
18
Example artificial models
2. Auto-Regressive (2) series
19
(No Transcript)
20
e.g. Call centre percentage of calls answered
within 30 secs
e.g. Production Line Manufacturing Plant
through-put / hour
REAL MODELS Models created in real
circumstances that cover each general type of
model and output encountered in real life
modeling.
e.g. Fast Food Store average queuing time
e.g. Swimming Pool complex average number in
system
Steady State
Transient
Steady State Cycle
With or without warm-up
Trend
21
Example real models
1. Argos Number of customers in queue to pay
Stochastic model with changing arrival rates.
Empty to empty transient autocorrelated
non-normal output.
22
Example real models
2. Leggings Manufacturing Plant Through-put /
hour
Stochastic model. Steady state with warm-up not
autocorrelated normal output.
23
Example real models
3. Sanitory Towel Packing Plant Through-put /
hour
Stochastic model with changing productivity in
work stations. Steady state

daily cycle.
Series of means of each cycle autocorrelated
non-normal output.
24
The AutoSimOA Project
Create a standard set of model outputs for
testing output analysis methods.

• We will use this representative and sufficient
  set of models/output when
• determining the most appropriate methods for
  automating simulation output analysis
• determining the effectiveness of the analysis
  methods
• revising the methods where necessary in order to
  improve their effectiveness and capacity for
  automation
• In order to propose a procedure for automated
  output analysis of warm-up, replications and
  run-length.

25
Automating the Analysis of Simulation Output Data
Thank you for listening.

• Stewart Robinson, Katy Hoad,
• Ruth Davies
• OR48, September 2006