Experimental Design, Response Surface Analysis, and Optimization

1
Experimental Design, Response Surface Analysis,
and Optimization

2
Outline

• Motivation and Terminology
• Difficulties in Solving the Basic Problem
• Examples of Factors and Responses
• Types/Examples of Experimental Design
• Full Factorial Designs
• Randomness of Effects
• Example Full Factorial Design
• Situations with Many Factors
• Response Surfaces and Metamodels
• Regression Analysis
• Response Surface Methodology

3
Motivation and Terminology

• Useful when there are many alternatives to
  consider (e.g., numerous capacity levels of
  various types, numerous parameters for a proposed
  inventory system)
• Two basic types of variables factors and
  responses
• Factors input parameters
• -controllable or uncontrollable
• -quantitative or qualitative
• Responses outputs from the simulation model
• -uncertain in nature
• Basic problem find the best levels (or values of
  the parameters) in terms of the responses
• Experimental Design can tell you which
  alternatives to simulate so that you obtain the
  desired information with the least amount of
  simulation

4
Difficulties in Solving the Basic Problem

• Multiple Responses
• Uncertain Responses

5
Examples of Factors

• 1. Mean interarrival time (uncontrollable,
  quantitative)
• 2. Mean service time (controllable or
  uncontrollable, quantitative)
• 3. Number of servers (controllable,
  quantitative)
• 4. Queuing discipline (controllable,
  qualitative)
• 5. Reorder point (controllable, quantitative)
• 6. Mean interdemand time (uncontrollable,
  quantitative)
• 7. Distribution of interdemand time
  (uncontrollable, qualitative)

6
Examples of Responses

• 1. Mean daily production rate
• 2. Mean time in the system for patients
• 3. Mean inventory level
• 4. Number of customers who wait for more than 5
  minutes

7
Types/Examples of Experimental Designs

• Completely Randomized Designs
• Randomized Complete Block Designs
• Nested Factorial Designs
• Split Plot Type Designs
• Latin Square Type Designs
• Full Factorial Designs
• Fractional Factorial Designs

8
2k Factorial Designs

• Suppose that we have k (k gt 2) factors. A 2k
  factorial design would require that two levels be
  chosen for each factor, and that n simulation
  runs (replications) be made at each of the 2k
  possible factor-level combinations (design
  points). For 3 factors, this yields a Design
  Matrix

9
2k Factorial Designs

• Design Matrix for 3 Factors
• Factor 1 Factor 2 Factor 3
• Design Point Level Level
  Level Response
• 1 - - - O1
• 2 - - O2
• 3 - - O3
• 4 - O4
• 5 - - O5
• 6 - O6
• 7 - O7
• 8 O8
• refers to one level of a factor and -
  refers to the other level. Normally, for
  quantitative factors, the smallest and largest
  levels for each factor are chosen.

10
2k Factorial Designs Estimating Main Effects

• The main effect of factor 1 is the change in the
  response variable as a result of the change in
  the level of the factor, averaged over all levels
  of all of the other factors
• If the effect of some factor depends on the level
  of another factor, these factors are said to
  interact.
• The degree of interaction (two-factor interaction
  effect) between two factors i and j is defined as
  half the difference between the average effect of
  factor i when factor j is at its level ( and
  all factors other than i and j are held constant)
  and the average effect of i when j is at its -
  level for example,

11
Example Full Factorial (2k) Design

• Consider a simulation model of reorder point,
  reorder quantity inventory system. The two
  decision variables, or factors, to consider are
  the reorder point (P) and the reorder quantity
  (Q) for the inventory system. The maximum and
  minimum allowable values for each are given
  below
• Suppose that the response variable output by
  the model is the long-run average monthly cost
  (composed of three components holding cost,
  shortage costs, and ordering costs) in thousands
  of dollars.

12
Example Full Factorial (2k) Design

• A 22 factorial design matrix with simulation
  results (for 10 replications at each design
  point) might be given by
• where a factor level of - indicates the
  minimum possible value for that factor, and a
  factor level of indicates the maximum
  possible level for example, design point 2 has
  P40 and Q15.

13
Example Full Factorial (2k) Design

• The response given is the average cost over the
  10 replications. Now, the main effects are given
  by
• The interaction effect (ePQ) is given by
• Therefore, the average effect of increasing P
  from 20 to 40 is to increase monthly cost by 1.2,
  and the average effect of increasing Q from 15 to
  50 is to decrease monthly cost by 5.3. Hence, it
  would be advisable to set P as low as possible
  and set Q as high as possible.

14
Example Full Factorial (2k) Design

• Also since the interaction effect is positive, it
  would seem advisable to set P and Q at opposite
  levels. (Of course, all of the above could be
  inferred from a cursory analysis of the responses
  for the various design points.) Note also that
  the literal interpretation of main effects
  assumes no interaction effects (pages 669 and 670
  of Law and Kelton, 1991).

15
Randomness of Effects

• Note that the main and interaction effects
  computed in the previous examples are just random
  variables. To determine if the effects are
  significant or real, and not due to random
  fluctuations, one could compute values for the
  main and interaction effects 10 times (once for
  each replication) and form confidence intervals
  for each of the main effects, and the
  interactions effect. If the confidence interval
  contains 0, then the effect is not statistically
  significant. (Note that statistical significance
  does not necessarily imply practical
  significance).

16
Situations with Many Factors

• When there are many factors to consider, full
  factorial, or even fractional factorial designs
  may not be feasible from a computational
  standpoint.
• Other types of design (e.g., Plackett-Burman
  designs or supersaturated designs) may be
  appropriate (Mauro, 1986).
• Another approach is to reduce the number of
  factors to consider via factor screening
  techniques, involving, for example, group
  screening in which a whole group is treated as a
  factor.

17
Response Surfaces and Metamodels

• A response surface is a graph of a response
  variable as a function of the various factors.
• A metamodel (literally, model of the simulation
  model), is an algebraic representation of the
  simulation model, with the factors as independent
  variables and a response as the dependent
  variable. It represents an approximation of the
  response surface.
• The typical metamodel used in a simulation
  application is a regression model.

18
Response Surfaces and Metamodels

• A metamodel through the use of response surface
  methodology can be used to find optimal values
  for a set of factors. It can also be used to
  answer what if questions. (Experimentation
  with a metamodel is typically much less expensive
  than using a simulation model directly).
• An experimental design process assumes a
  particular metamodel, e.g.,

19
Basic Concepts of Regression Analysis

• Regression is used to determine the best
  functional relation among variables.
• Suppose that the functional relationship is
  represented as
• E(Y) f (X1, ..., Xp / B1, ..., BE)
• where E(Y) is the expected value of the response
  variable Y the X1, ..., Xp are factors
  and the B1, ..., BE are function
  parameters e.g.,
• E(Y) B1 B2 X1 B3 X2 B4 X1 X2

20
Basic Concepts of Regression Analysis

• The observed value for Y, for a given set of X
  s, is assumed to be a random variable, given by
• Y f (X1, ..., Xp/B1, ..., BE)
• Where is a random variable with mean equal
  to 0 and variance . The values for
  B1,...,BE are obtained by minimizing the sum of
  squares of the deviations.

21
Response Surface Methodology

• Source (Fu, 1994)
• Response surface methodology (RSM) involves a
  combination of metamodeling (i.e., regression)
  and sequential procedures (iterative
  optimization).

22
Response Surface Methodology

• RSM involves two phases
• Fit a linear regression model to some initial
  data points in the search space (through
  replications of the simulation model). Estimate
  a steepest descent direction from the linear
  regressions model, and a step size to find a new
  (and better) solution in the search space.
  Repeat this process until the linear regression
  model becomes inadequate (indicated by when the
  slope of the linear response surface is
  approximately 0 i.e., when the interaction
  effects become larger than than the main
  effects).
• Fit a nonlinear quadratic regression equation to
  this new area of the search space. Then find the
  optimum of this equation.

23
Terminology in Experimental Design

• Source(Ostle, 1963)
• Replication - the repetition of the basic
  experiment
• Treatment - a specific combination of several
  factor levels
• Experimental Unit - the unit to which a single
  treatment is applied to one replication of the
  basic experiment
• Experimental Error - the failure of two
  identically treated experimental units to yield
  identical results
• Confounding - the mixing up of two or more
  factors so that its impossible to separate the
  effects

24
Terminology in Experimental Design

• Randomization - randomly assigning treatments to
  experimental units (assures independent
  distribution of errors)
• Main Effect (of a factor) - a measure of the
  change in a response variable to changes in the
  level of the factor averaged over all levels of
  all the other factors
• Interaction is an additional effect (on the
  response) due to the combined influence of two or
  more factors