Pitfalls

1
Pitfalls
2
No problem definition
"We have many problems in our line of
business. Just start building a simulation model
of our business while we figure out which
questions we want to be answered."
When the questions arrive (if at all answerable
by simulation), the model cannot be used.
Too much details complex simulation model many
parameters to assess many errors to
correct lengthy simulation runs
Too few details not enough information at best
only partial answers
3
One-sided problem definition
"What investments do I have to make to keep my
customers satisfied?"
Clarify possible investments, quality of
service, assumptions about demand.
Both client and consultant should understand and
agree.
understand
make formal and explain!
Problem definition simulation contract.
4
Hidden assumptions
Many assumptions are made for a simulation study.
Keep track of assumptions made. "In DCT case,
total crane service time of an X-type
truck succeeding an Y-type truck is normally
distributed with average a(X,Y) and standard
deviation s(X,Y)." If possible, validate (compare
with recorded trace). In all cases mention in
report.
Hidden assumptions used in model but not
reported.
Hidden assumptions may lead to costly mistakes!
5
Force fit the model
Tandem queue arrival expo(1), p1, p2
triangle(0.5,0.9,1.3) 10 subruns of 1000 initial
run of 10 measured queue sizes p1 1.71, p2 0.42
First model correct average but variance too
small arrival expo(1), p1, p2
triangle(0.7,0.8,0.9) measured queue sizes p1
1.56, p2 0.11
Fiddle with average to make model fit p1
triangle(0.71,0.81,0.91), p2 triangle(0.735,0.835
,0.935) measured queue sizes p1 1.78, p2 0.42
Analyze 11 increased demand. 3.5/0.85 versus
4.6/3.9!
6
Improper use of statistics
Giving Statistical Packages to Engineers is like
Giving Guns to Children, Dick Mensing
"Statistics don't lie, but liars use
statistics" Darell Huff
7
Methodology pitfall
Simulation Experimentation. Design of
experiments established rules. Hypothesis before
experiment. New hypothesis new experiment.
Hypothetical case beer appreciation. Brewers
have designed "X", a new low-cost
beer. Hypothesis appreciation of "X" not lower
than other beers. Consumer appreciation is tested
in various Dutch cities, relatively to "Amstel"
and "Grolsch".
8
Beer appreciation
Table of average appreciations per city.
city X Ams Grol A'dam 6.5 7.2
6.9 R'dam 7.1 6.8 7.3 den Haag 6.3 6.5
7.1 Utrecht 5.7 6.1 6.2 E'hoven 7.1 6.9
7.0 Enschede 5.9 6.6 7.5 Gr'ingen 6.2 6.3
6.5 Tilburg 6.9 7.0 6.7 Haarlem 6.8 7.2
6.8 Breda 7.2 6.9 7.2 Nijmegen 7.0 7.1 7.3
city X Ams Grol
E'hoven 7.1 6.9 7.0
Tilburg 6.9 7.0 6.7
Breda 7.2 6.9 7.2

AVG 7.07 6.93 6.97
Statistical analysis with 98 confidence, people
from N-Brabant prefer (on average) "X" above the
other two.
AVG 6.61 6.78 6.95
Hypothesis rejected.
9
Beer marketing strategy
Market "X" as a local beer in N-Brabant. Call it
"Brabobier". Make a lot of money.
What is wrong?
With 98 confidence, people from N-Brabant prefer
"X" above the other two.
New hypothesis based on sample observation.
Confidence cannot be assessed.
New, independent experiment is needed.
10
Observe random sample
Dice throwing experiment (Arena) 25 throws
sequence 2321116466156134222666665
five equals in a row! x(19)...x(24)6
probability 0.00013 twenty possibilities for
first 0.0026 five equals (instead of sixes)
0.0156
Hypothesis by throwing a dice 25 times, five
equals in row will occur. Confirmed with 98.5
confidence (on given sample).
Any recorded sample contains remarkable data. You
need a second sample to confirm a
hypothesis based on observing the first sample.
11
Simulation analogon
Problem in business process (e.g. DCT). Assess
tentative solution through simulation. Hypothesis
performance indicator x after implementing
the solution will be lower on average than in
current situation. Boss needs 90 confidence in
order to start implementing it.
Indicator Average Half Width x 4.18 0.79
current
30 subruns of 1000 hrs
tentsol
x 3.89 0.85
Test hypothesis evaluate
C
T
result 0.73
12
Longer simulation
Indicator Average Half Width x 4.27 0.41
current
30 subruns of 4000 hrs
tentsol
x 3.92 0.47
hypothesis confidence 0.87
Indicator Average Half Width x 4.32 0.38
current
30 subruns of 6000 hrs
tentsol
x 3.91 0.41
outcome 0.93
?
conclusion with 93 reliability solution gives
lower average x
13
Persistent experimentation
Repeat an experiment until it satisifies
hypothesis. Statistically analyze each experiment
individually.
Hypothesis corroboration with incorrect
confidence!
After 3 experiments goal of 90 confidence
attained.
A new, independent simulation run is needed to
confirm! Choose different initial value for
random generator and repeat the last experiment.
14
Cheating recipes
Confirm hypothesis by simulation with 95
reliability. Do many independent simultation
experiments. Only retain favorable ones. Claim
you did not execute the unfavorable ones.
Speeding up the process Perform e.g. 50
iterations. Look for a most favorable sequence of
30, e.g. 7-36. Define iterations 0-6 as initial
run. Perform 30 iterations.
Round off favorably (model parameters and
simulation results)
15
Abuse of statistics
Recent uncovered evidence shows that nurse Lucia
de B has been convicted of murder on the basis of
statistical arguments only and by abusing just
about every rule in the book.   Whether or not
an incident was classified as suspect depended
on whether or not Lucia was on duty (simply a
question of checking which nurse is on duty and
then asking enough doctors till you get a
"suspicious" verdict. Data was collected in
this way till there was enough to condemn her.
A professor of statistics in law, and trained
mathematician, does not know the meaning of one
of the most basic statistical concepts - the
p-value. (Statistician xxx multiplied three
independent p-values in order to obtain a
combined p-value).It somehow reminds me of the
old method to see if someone is a witch - if they
drown they were innocent, if they are guilty you
can burn them.
16
Quote
"The only way to have real success in
science...is to describe the evidence very
carefully without regard to the way you feel it
should be." Richard Feynman