Verification

1
Verification Validation

• Jerry Banks

2
Verification

• Did we build the model right?
• Is the model implemented correctly in the
  computer?

3
Validation

• Did we build the right model?
• Determination that the simulation model is an
  accurate representation of the system for the
  objectives of the study

4
All models are wrong. Some are useful.
George Box
Keep the first part in mind so as to maximize the
likelihood of the second part being true.
5
Models are guilty until proven innocent!
John Carson
6
Avoid the following syndrome
If the model runs to completion, it must be right.
7
Real system
Calibration validation
Conceptual validation
Conceptual model Assumptions on system
components. Structural assumptions that define
interactions between system components. Input
parameters and data assumptions
Model verification
Operational model (Computerized representation)
8
Verification Common sense suggestions

• Have the computerized representation checked by
  someone other than its developer

9
Verification Common sense suggestions

• Make a flow diagram which includes each logically
  possible action that a system can take when an
  event occurs, and follow the model logic for each
  event type

10
Verification Common sense suggestions

• Closely examine the model output for
  reasonableness under a variety of settings of the
  input parameters
• Have the computerized representation print a wide
  variety of output statistics

11
Verification Common sense suggestions

• Have the computerized representation print the
  input parameters at the end of the simulation to
  be sure that they have not been changed
  inadvertently and that they agree with the input
  that was intended

12
Verification Common sense suggestions

• Make the computerized representation as
  self-documenting as possible
• Give a precise definition of every variable used
  and a general description of the purpose of each
  major section of code

13
Verification Common sense suggestions

• Verify that what is seen in the animation
  imitates the actual system

14
Verification Common sense suggestions

• Use the interactive run controller
• Monitor the system as it progresses
• Focus on a particular line or section of logic
• Observe values of selected model components
• Pause the simulation and change input parameters

15
Faulty indexing is the number one source of
errors
Jim Henriksen
16
Validation

• If the model is valid, it can be used to make
  decisions about the real system
• Validation is much easier if a version of the
  system exists
• Absolute validation is not possible
• A model valid for one objective might not be
  valid for another objective

17
Validation

• Validation should be performed throughout the
  development of the simulation model
• Not performed at the end of model building if
  there are funds available!

18
Practical suggestions

• Construct models that have high face validity
• They appear reasonable to knowledgeable persons

19
Practical suggestions

• Formulate the problem precisely
• Include a definitive list of the questions to be
  answered
• This aids in determining the level of detail
  needed
• As more is learned, the problem may be
  reformulated

20
Practical suggestions

• No one person knows everything
• So, talk to many people

21
Practical suggestions

• Interact with the decision-maker regularly
• Helps avoid a g error
• Keeps the decision-maker involved
• Aids credibility as the decision-maker
  understands the model and the assumptions

22
Practical suggestions

• Document
• Assumptions, algorithms, programs, input data
  summaries, etc.
• Include a CAD drawing
• Detailed description of each subsystem

23
Practical suggestions

• Structured walk-through of the conceptual model
• Include subject matter experts and
  decision-makers
• Go through the conceptual model bullet-by-bullet
• Make sure that everyone agrees with the bullet
  and level of detail

24
Practical suggestions

• Determine which inputs are important
• If the model is insensitive to a parameter, dont
  waste time getting it exactly right
• If the model is insensitive to a distribution,
  dont waste time getting it exactly right

25
Practical suggestions

• Force rare events to see how the model reacts

26
Practical suggestions

• Use timelines instead of relying solely on
  summary statistics
• See if resources are being used

27
Practical suggestions

• Examine a wide variety of output measures
• More than primary measures
• All queues
• More than throughput
• Throughput Input

28
The validation dream

• The models output matches that of the actual
  system

29
Example using paired t-test
30
Example, continued

• dbar2 5,343.2
• Sd2 7.580 x 107
• H0 md 0
• t0 dbar/Sd/SQRT(n)
• t0 5343.2/8705.85/SQRT(5) 1.37
• ta/2,n-1 t.025,5 2.78
• Reject the null hypothesis at a .05

31
When all else fails

• Use a Turing test
• Persons knowledgeable about system behavior can
  be used to compare model output to system output
• Five reports of system performance over five
  different days
• Simulation output data are used to produce five
  fake reports
• See if a knowledgeable person can decide which is
  which

32
End