Validation and Verification of Simulation Models

1
Validation and Verification of Simulation Models
2
Outline

• Introduction
• Definitions of validation and verification
• Techniques for verification of simulation models
• Techniques for validation of simulation models
• Statistical Methods for Comparing real-world
  observations with simulation output data
• Inspection Approach
• Confidence-Interval Approach
• Summary

3
Introduction

• One of the most difficult problems facing the
  simulation analyst is determining whether a
  simulation model is an accurate representation of
  the actual system being studied ( i.e.,
  whether the model is valid).
• If the simulation model is not valid, then any
  conclusions derived from it is of virtually no
  value.
• Validation and verification are two of the most
  important steps in any simulation project.

4
What are Validation and Verification?

• Validation is the process of determining whether
  the conceptual model is an accurate
  representation of the actual system being
  analyzed. Validation deals with building the
  right model.
• Verification is the process of determining
  whether a simulation computer program works as
  intended (i.e., debugging the computer program).
  Verification deals with building the model right.

Validation
Conceptual Model
Real -World System
Verification
Validation
Simulation Program
5
Techniques for Verification of Simulation Models

• Use good programming practice
• Write and debug the computer program in modules
  or subprograms.
• In general, it is always better to start with a
  moderately detailed model, and later embellish,
  if needed.
• Use structured walk-through
• Have more than one person to read the
  computer program.
• Use a trace
• The analyst may use a trace to print out some
  intermediate results and compare them with hand
  calculations to see if the program is
  operating as intended.

6
Techniques for Verification of Simulation Models

• Check simulation output for reasonableness
• Run the simulation model for a variety of input
  scenarios and check to see if the output is
  reasonable.
• In some instances, certain measures of
  performance can be computed exactly and used for
  comparison.
• Animate
• Using animation, the users see dynamic displays
  (moving pictures) of the simulated system.
• Since the users are familiar with the real
  system, they can detect programming and
  conceptual errors.

7
Techniques for Verification of Simulation Models

• Compare final simulation output with analytical
  results
• May verify the simulation response by running a
  simplified version of the simulation program with
  a known analytical result. If the results of the
  simulation do not deviate significantly from the
  known mean response, the true distributions can
  then be used.
• For example, for a queuing simulation model,
  queuing theory can be used to estimate steady
  state responses (e.g., mean time in queue,
  average utilization). These formulas, however,
  assume exponential interarrival and service times
  with n servers (M/M/n).

8
Techniques for Validation of Simulation Models

• A three-step approach for developing a valid and
  credible model
• 1. Develop a model with high face validity
• The objective of this step is to develop a
  model that, on the surface, seems reasonable to
  people who are familiar with the system under
  study.
• This step can be achieved through discussions
  with system experts, observing the system, or the
  use of intuition.
• It is important for the modeler to interact
  with the client on a regular basis throughout
  the process.
• It is important for the modeler to perform a
  structured walk-through of the conceptual
  model before key people to ensure the
  correctness of models
  assumptions .

9
Techniques for Validation of Simulation Models

• 2. Test the assumptions of the model empirically
• In this step, the assumptions made in the initial
  stages of model development are tested
  quantitatively. For example, if a theoretical
  distribution has been fitted to some observed
  data, graphical methods and goodness of fit tests
  are used to test the adequacy of the fit.
• Sensitivity analysis can be used to determine if
  the output of the model significantly changes
  when an input distribution or when the value of
  an input variable is changed. If the output
  is sensitive to some aspect of the
  model, that aspect of the model must be modeled
  very carefully.

10
Techniques for Validation of Simulation Models

• 3. Determine how representative the simulation
  output data are
• The most definitive test of a models validity is
  determining how closely the simulation output
  resembles the output from the real system.
• The Turing test can be used to compare the
  simulation output with the output from the real
  system. The output data from the simulation can
  be presented to people knowledgeable about the
  system in the same exact format as the system
  data. If the experts can differentiate between
  the simulation and the system outputs, their
  explanation of how they did that should improve
  the model.
• Statistical methods are available for comparing
  the output from the simulation model with those
  from the real-world system .

11
Statistical Methods for Comparing Real-World
Observation With Simulation Output Data

• Suppose are observations
  from a real-world system and
  are output data from
  the simulation model.
• Both, the real-world system outputs and
  simulation outputs are almost always
  non-stationary (the distribution of the
  successive observations change over time) and
  autocorrelated(the observations are correlated
  with each other).
• Therefore, because classical statistical tests (
  the t-test, chi-square test, K-S test, etc.)
  assume I.I.D data, they can not directly be used
  to compare the two data sets to determine model
  validity.

12
Statistical Methods for Comparing Real-World
Observation With Simulation Output Data

• 2 approaches for comparing the outputs from the
  real-world system with the simulation outputs
  are
• Inspection Approach
• Confidence-Interval Approach

13
Inspection Approach

• Run the simulation model with historical system
  input data (e.g., actual observed interarrival
  and service times) instead of sampling from the
  input probability distributions, and compare the
  system and model output data.
• The system and the model experience exactly the
  same observations from the input random
  variables.
• This approach results in model and system outputs
  being positively correlated.

14
Inspection Approach
Historical system input data
Historical system input data
Simulation model
Actual system
System output data
Model output data
15
Inspection Approach

• If X is the output from the real-world system and
  Y is the corresponding output from the model, we
  are interested in estimating .
• We make n experiments (using historical data) and
  compute Xj - Yj (for j 1, 2, , n) as an
  estimate of . Note that Xj and Yj
  use exactly the same interarrival times and
  service times).
• Xj and Yj can then be plotted such that the
  horizontal axis denotes time and the vertical
  axis denotes the real and simulated outputs. The
  user can then eyeball timepaths to see if the
  model accurately represents the real-world
  system.

16
Inspection Approach

• Due to positive correlation between X and Y,
  Var (X-Y) is much smaller than if X and Y were
  independent which makes Xj - Yj a much better
  estimate of
• Reason
• -- In general,
• Var (X-Y) Var (X) Var (Y) - 2Cov(X, Y)
• -- If X and Y are independent, Cov (X, Y)0
• and Var (X-Y) Var (X) Var (Y)
• -- But, if X and Y are positively correlated,
  Cov(X, Y) gt 0 leading to a smaller value for Var
  (X-Y).

17
Inspection Approach

• Inspection approach (also called trace driven
  method)may provide valuable insight into the
  adequacy of the simulation model for some
  simulation studies.
• In fact, this may be the only feasible approach
  because of severe limitations on the amount of
  data available on the operation of some
  real-world systems (e.g., military situation).
• Care must, however, be taken in interpreting the
  result of this approach.

18
Confidence-Interval Approach

• A more reliable approach for comparing a
  simulation model with the real-world system.
• Requires a large amount of data.
• Suppose we collect m independent sets of data
  from the system and n independent sets of data
  from the model (m and n can be equal).
• Let Xj be the average of observations in the jth
  set of system data with mean and let Yj be
  the output from the jth replication of the
  simulation model with .
• The objective is to build a confidence- interval
  for

19
Confidence-Interval Approach

• In the case of correlated outputs where Xj is
  correlated with Yj (e.g., using trace driven
  simulation), let m n and pair Xj s and Yjs.
  Let Zj Xj -Yj for j1, 2 , n. Zjs are IID
  random variables with E(Zj) .
• Let
• and
• Then, the approximate 100(1- ) percent C.I. is
• If the confidence interval does not include a
  zero, then the observed difference between
  and is statistically different at level
  .

20
Summary

• Validation and verification are two of the most
  important steps in any simulation study.
• Validation is not something to be attempted after
  the simulation model has already been developed,
  and only if there is time and money still
  remaining. Instead, model development should be
  done throughout the entire simulation study.