SAE 599 Modeling and Simulation for Systems Architecting and Engineering

1
SAE 599 - Modeling and Simulation for Systems
Architecting and Engineering

• Dr. Raymond Madachy
• November 21, 2007

2
Outline

• Test problem review
• Analysis of simulation output
• Confidence intervals
• Homework and readings

3
Review Combined Discrete and Continuous
Simulation

• Types of interactions between discretely changing
  and continuously changing variables
• A discrete event causes a discrete change in the
  value of a continuous state variable
• A discrete event causes a relationship governing
  a continuous state variable to change at a
  particular time
• A continuous state variable achieving a threshold
  value causes a discrete event to occur or to be
  scheduled

4
Planetary Rover Combined Simulation Review
5
Analysis of Simulation Output

• Simulation models convert stochastic inputs and
  system components into statistical data output
• hence, simulation is another sampling method and
  the output is subject to statistical analysis
• Variable estimates are subject to sampling choice
  and sample size
• determining proper sample size may require some
  knowledge of parameters to estimate
• conclusions should not be based on results of
  single simulation run
• System Classification for Output Analysis
• Non-terminating
• have no end to operations for practical time
  horizon
• e.g. phone system, emergency room, traffic, even
  factories when daily start condition is same as
  end of previous day
• usually, but not always have steady state(s)
• may wish to study transient or steady state
  conditions
• treat as terminating system to study transient
  state

6
Analysis of Simulation Output (cont.)

• System Classification for Output Analysis
• Terminating systems
• usually start from no-action or empty state and
  end with either of same termination occurs after
  time delta or event occurrence
• e.g. time lapse termination bank hours,
  quarterly inventory
• event termination device failure, bid process,
  war
• may or may not reach steady state
• if steady state exists, system may be treated as
  non-terminating for certain purposes
• Several runs are necessary if event-controlled
  terminating systems don't reach steady state
• use different random number allocation per
  replication
• Data Dependency
• Independency of sample data allows use of
  classical statistical analysis
• However, most discrete event simulation models
  lack independency
• queueing process usually leads to autocorrelation
  
• may choose to select samples far apart in the
  system

7
Analysis of Simulation Output (cont.)

• Independent Replications
• Addresses problem of autocorrelation. Statistical
  techniques covered so far assume samples are
  independent and identically distributed (IID).
• perform several short runs (replications) from
  time0
• use different random number seeds per run (or
  possibly different initial conditions)
• replications are then independent of each other
• each replication mean is an independent
  observation
• Compute mean, variance and confidence interval
• Batch Means
• Method only applies to steady state analysis
• alleviates problem of handling transitional
  periods during independent replications
• divide single run into multiple intervals
  (batches)
• shorter intervals impose stronger dependencies,
  so larger batch sizes and fewer batches are
  recommended
• batch mean values serve as independent samples
• compute each mean and the grand mean

8
Confidence Intervals

• Estimate accuracy expressed as a confidence
  interval (CI)
• CI interpretation if one constructs a very large
  number of 1-a CIs each based on n observations,
  in the long run 1- a of the intervals contain the
  parameter value. a is also called the
  significance level, or the probability of
  rejecting a hypothesis when it's true.
• for given confidence level, a small confidence
  interval is preferable
• for given confidence interval, a higher
  confidence level is preferable
• in practice, choose a confidence level
• sample size affects confidence level and interval
  
• smaller confidence intervals with larger sample
  size
• estimation of mean
• due to central limit theorem, distribution of
  sample mean is normal
• Z is normally distributed
• Compute 1-a CI. Use normal distribution tables if
  gt30 samples, or student t-distribution if lt 30
  samples.
• Other measures for confidence intervals
• estimation of proportion
• estimation of difference between means

9
Confidence Intervals (cont.)

• Confidence interval equation
• where X is the sample mean, m is the population
  mean and s is the standard deviation
• Below is a normal distribution with a mean of
  one. The true mean falls in the confidence
  interval with a probability of (1-a).

10
Confidence Intervals (cont.)

• Sample Size Selection
• can use desired CI width in conjunction with
  estimated sample variance to determine proper
  sample size
• Hypothesis Testing
• start with a null hypothesis and set the
  significance level a (CI is 1- a)
• E.g., to test whether population mean equals a
  given value fail to reject the hypothesis if
  confidence intervals contains value

11
Confidence Intervals (cont.)

• From Monte Carlo example with16 iterations, we
  use a t-distribution table and find a value of
  1.75 corresponding to 15 degrees of freedom and
  t95. Since we want a 90 interval, a/25 and we
  keep 5 at each tail at the distribution by
  looking up the value for 1-a/2. Thus the 90
  confidence interval is
• P 238 1.7558.8/?16lt m lt 238
  1.7558.8/?16 .9 212 lt m lt 264
• Thus the 90 confidence interval for simulated
  effort is between 212 and 264 person-months, and
  we fail to reject the hypothesis that the means
  lies in that interval.

12
Verification and Validation Redux

• Verification - is model implementation error-free
  and properly represent intended logical behavior?
  
• problems may be due to coding or logic errors
• possible sources of error
• numerical data errors
• unexpected random variates
• inconsistent units
• variable overwriting
• concurrent event processing
• entity flow problems
• deadlocks
• incorrect statistics specifications
• error prevention approaches
• modular validation
• readable and commented programs
• try alternative approaches
• outside analysts
• numerical evaluations
• induce infrequent events
• event animations

• Validation - does model help solve end-user's
  problem?
• must consider within context of study purpose
• approaches for checking validity
• parameter and relationship testing
• structural and boundary testing
• sensitivity analysis
• continuity testing
• degeneracy testing
• testing extreme or absurd conditions

13
Test Topic Coverage (cont.)

• Homework
• Complete Brookss Law model homework
• Due 11/28
• Can wait until 12/5 if you present on 11/28
• Reading
• Law-Kelton Textbook Chapters 9 and 10