Simulation Modeling and Analysis

1
Simulation Modeling and Analysis

• Output Analysis

1
2
Outline

• Stochastic Nature of Output
• Taxonomy of Simulation Outputs
• Measures of Performance
• Point Estimation
• Interval Estimation
• Output Analysis in Terminating Simulations
• Output Analysis in Steady-state Simulations

2
3
Introduction

• Output Analysis
• Analysis of data produced by simulation
• Goal
• To predict system performance
• To compare alternatives
• Why is it needed?
• To evaluate the precision of the simulation
  performance parameter as an estimator

3
4
Introduction -contd

• Each simulation run is a sample point
• Attempts to increase the sample size by
  increasing run length may fail because of
  autocorrelation
• Initial conditions affect the output

4
5
Stochastic Nature of Output Data

• Model Input Variables are Random Variables
• The Model Transforms Input into Output
• Output Data are Random Variables
• Replications of a model run can be obtained by
  repeating the run using different random number
  streams

5
6
Example M/G/1 Queue

• Average arrival rate Poisson with ? 0.1 per
  minute
• Service times Normal with ? 9.5 minutes and
  ? 1.75 minutes
• Runs
• One 5000 minute run
• Five 1000 minute runs w/ 3 replications each

6
7
Taxonomy of Simulation Outputs

• Terminating (Transient) Simulations
• Runs until a terminating event takes place
• Uses well specified initial conditions
• Non-terminating (Steady-state) Simulations
• Runs continually or over a very long time
• Results must be independent of initial data
• Termination?
• What determines the type of simulation?

7
8
Examples Non-terminating Systems

• Many shifts of a widget manufacturing process.
• Expansion in workload of a computer service
  bureau.

8
9
Measures of Performance Point Estimation

• Means
• Proportions
• Quantiles

9
10
Measures of Performance Point Estimation
(Discrete-time Data)

• Point estimator of ?? (of ?) based on the
  simulation discrete-time output
  (Y1, Y2,.., Yn)
• ? (1/n) ? i n Yi
• Unbiased point estimator
• E(? ) ?
• Bias
• b E(? ) - ?

10
11
Measures of Performance Point Estimation
(Continuous-time data)

• Point estimator of ?? (of ?) based on the
  simulation continuous-time output
  (Y(t), 0 lt t lt Te)
• ? (1/ Te) ? 0 Te Y(t) dt
• Unbiased point estimator
• E(? ) ?
• Bias
• b E(? ) - ?

11
12
Measures of Performance Interval Estimation
(Discrete-time Data)

• Variance and variance estimator
• ?2(??) true variance of point estimator ???
• ?2(??) estimator of variance of point
  estimator ??
• Bias (in variance estimation)
• B E(?2(??) )/ ?2(??)

12
13
Measures of Performance Interval Estimation -
contd

• If B 1 then t (?? - ?)/ ?2(??) has t?/2,f
  distribution (d.o.f. f). I.e.
• A 100(1 - ?) confidence interval for ??is
• ?? - t?/2,f ?2(??) lt ? lt ?? t?/2,f
  ?2(??)
• Cases
• Statistically independent observations
• Statistically dependent observations (time
  series).

13
14
Measures of Performance Interval Estimation -
contd

• Statistically independent observations
• Sample variance
• S2 ? i n (Yi - ?? )2/(n-1)
• Unbiased estimator of ?2(??)
• ?2(??) S2 /n
• Standard error of the point estimator ??
• ?(??) S /?n

14
15
Measures of Performance Interval Estimation -
contd

• Statistically dependent observations
• Variance of ??
• ?2(??) (1/n2) ? i n ? j n cov(Yi , Yj )
• Lag k autocovariance
• ?k cov(Yi , Yik )
• Lag k autocorrelation
• ?k ?k??0

15
16
Measures of Performance Interval Estimation -
contd

• Statistically dependent observations (contd)
• Variance of ??
• ?2(??) (?0 /n) 1 2? k1 n-1 (1- k/n) ?k
  (?0 /n) c
• Positively autocorrelated time series (?k gt 0)
• Negatively autocorrelated time series (?k lt 0)
• Bias (in variance estimation)
• B E(S2/n )/ ?2(??) (n/c - 1)/(n-1)

16
17
Measures of Performance Interval Estimation -
contd

• Statistically dependent observations (contd)
• Cases
• Independent data ?k 0, c 1, B 1
• Positively correlated data ?k gt 0, c gt 1, B lt 1,
  S2/n is biased low (underestimation)
• Negatively correlated data ?k lt 0, c lt 1, B gt 1,
  S2/n is biased high (overestimation)

17
18
Output Analysis for Terminating Simulations

• Method of independent replications
• n Sample size
• Number of replications r1,2,,R
• Yji i-th observation in replication j
• Yji, Yjk are autocorrelated
• Yri, Ysk are statistically independent
• Estimator of mean (r 1,2,,R)
• ??r???(1/nr) ? i nr Yri

18
19
Output Analysis for Terminating Simulations -
contd

• Confidence Interval (R fixed discrete data)
• Overall point estimate
• ? (1/R) ? 1 R ??r?
• Variance estimate
• ?? (?) 1/(R-1)R ? 1 R (??r?????????
• Standard error of the point estimator ??
• ?(??) ? ?? (?)

19
20
Output Analysis for Terminating Simulations -
contd

• Estimator and Interval (R fixed continuous data)
• Estimator of mean (r 1,2,,R)
• ??r???(1/Te) ? 0 Te Yr(t) dt
• Overall point estimate
• ? (1/R) ? 1 R ??r?
• Variance estimate
• ?? (?) 1/(R-1)R ? 1 R (??r?????????

20
21
Output Analysis in Terminating Simulations - contd

• Confidence Intervals with Specified Precision
• Half-length confidence interval (h.l.)
• h.l. t?/2,f ?2(??) t?/2,f S/ ?R lt ?
• Required number of replications
• R gt ( z ?/2 So/ ? )2

21
22
Output Analysis for Steady State Simulations

• Let (Y1, Y2,.., Yn) be an autocorrelated time
  series
• Estimator of the long run measure of performance
  ? (independent of I.C.s)
• ? lim n gt? (1/n) ? i n Yi
• Sample size n (or Te) is design choice.

22
23
Output Analysis for Steady State Simulations
-contd

• Considerations affecting the choice of n
• Estimator bias due to initial conditions
• Desired precision of point estimator
• Budget/computer constraints

23
24
Output Analysis for Steady State Simulations
-contd

• Initialization bias and Initialization methods
• Intelligent initialization
• Using actual field data
• Using data from a simpler model
• Use of phases in simulation
• Initialization phase (0 lt t lt To for i1,2,,d)
• Data collection phase (To lt t lt Te for
  id1,d2,,n)
• Rule of thumb (n-d) gt 10 d

24
25
Output Analysis for Steady State Simulations
-contd

• Example M/G/1 queue
• Batched data
• Batched means
• Averaging batch means within a replication (I.e.
  along the batches)
• Averaging batch means within a batch (I.e. along
  the replications).

25
26
Steady State Simulations Replication Method

• Cases
• 1.- Yrj is an individual observation from within
  a replication
• 2.- Yrj is a batch mean of discrete data from
  within a replication
• 3.- Yrj is a batch mean of continuous data over a
  given interval

26
27
Steady State Simulations Replication Method
-contd

• Sample average for replication r of all
  (nondeleted) observations
• Yr(n,d) Yr 1/(n-d) ? jd1n Yrj
• Replication averages are independent and
  identically distributed RVs
• Overall point estimator
• Y(n,d) Y 1/R ? r1R Yr(n,d)

27
28
Steady State Simulations Replication Method
-contd

• Sample Variance
• S2 1/(R-1) ? r1R (Yr - Y)
• Standard error S/ ?R
• 100(1-?) Confidence interval
• Y - t ?/2,R-1 S/ ?R lt ? lt Y t ?/2,R-1 S/ ?R

28
29
Steady State Simulations Sample Size

• Greater precision can be achieved by
• Increasing the run length
• Increasing the number of replications

29
30
Steady State Simulations Batch Means for
Interval Estimation

• Single, long replication with batches
• Batch means treated as if they were independent
• Batch means (continuous)
• Yj (1/m) ? (j-1)m jm Y(t) dt
• Batch means (discrete)
• Yj (1/m) ? i(j-1)m jm Yi

30
31
Steady State Simulations Batch Size Selection
Guidelines

• Number of batches lt 30
• Diagnose correlation with lag 1 autocorrelation
  obtained from a large number of batch means from
  a smaller batch size
• For total sample size to be selected sequentially
  allow batch size and number of batches grow with
  run length.

31