CHAPTER 17 OPTIMAL DESIGN FOR EXPERIMENTAL INPUTS

1
CHAPTER 17 OPTIMAL DESIGN FOR EXPERIMENTAL
INPUTS
Slides for Introduction to Stochastic Search and
Optimization (ISSO) by J. C. Spall

• Organization of chapter in ISSO
• Background
• Motivation
• Finite sample and asymptotic (continuous) designs
• Precision matrix and D-optimality
• Linear models
• Connections to D-optimality
• Key equivalence theorem
• Response surface methods
• Nonlinear models

2
Optimal Design in Simulation

• Two roles for experimental design in simulation
• Building approximation to existing large-scale
  simulation via metamodel
• Building simulation model itself
• Metamodels are curve fits that approximate
  simulation input/output
• Usual form is low-order polynomial in the inputs
  linear in parameters ?
• Linear design theory useful
• Building simulation model
• Typically need nonlinear design theory
• Some terminology distinctions
• Factors (statistics term) ? Inputs (modeling
  and simulation terms)
• Levels ? Values
• Treatments ? Runs

3
Unique Advantages of Design in Simulation

• Simulation experiments may be considered special
  case of general experiments
• Some unique benefits occur due to simulation
  structure
• Can control factors not generally controllable
  (e.g., arrival rates into network)
• Direct repeatability due to deterministic nature
  of random number generators
• Variance reduction (CRNs, etc.) may be helpful
• Not necessary to randomize runs to avoid
  systematic variation due to inherent conditions
• E.g., randomization in run order and input levels
  in biological experiment to reduce effects of
  change in ambient humidity in laboratory
• In simulation, systematic effects can be
  eliminated since analyst controls nature

4
Design of Computer Experiments in Statistics

• There exists significant activity among
  statisticians for experimental design based on
  computer experiments
• T. J. Santner et al. (2003), The Design and
  Analysis of Computer Experiments, Springer-Verlag
• J. Sacks et al (1989), Design and Analysis of
  Computer Experiments (with discussion),
  Statistical Science, 409435
• Etc.
• Above statistical work differs from experimental
  design with Monte Carlo simulations
• Above work assumes deterministic function
  evaluations via computer (e.g., solution to
  complicated ODE)
• One implication of deterministic function
  evaluations no need to replicate experiments for
  given set of inputs
• Contrasts with Monte Carlo, where replication
  provides variance reduction

5
General Optimal Design Formulation (Simulation or
Non-Simulation)

• Assume model
• z h(?,?x) v ,
• where x is an input we are trying to pick
  optimally
• Experimental design ? consists of N specific
  input values x ?i and proportions (weights) to
  these input values wi
• Finite-sample design allocates n ? N available
  measurements exactly asymptotic (continuous)
  design allocates based on n ? ?

6
D-Optimal Criterion

• Picking optimal design ? requires criterion for
  optimization
• Most popular criterion is D-optimal measure
• Let M(?,??) denote the precision matrix for an
  estimate of ? based on a design ?
• M(?,??) is inverse of covariance matrix for
  estimate
• and/or
• M(?,??) is Fisher information matrix for estimate
• D-optimal solution is

7
Equivalence Theorem

• Consider linear model
• Prediction based on parameter estimate and
  future measurement vector hT is
• Kiefer-Wolfowitz equivalence theorem states
• D-optimal solution for determining ? to be used
  in forming is the same ? that minimizes the
  maximum variance of predictor
• Useful in practical determination of optimal ?

8
Variance Function as it Depends on Input Optimal
Asymptotic Design for Example 17.6 in ISSO
9
Orthogonal Designs

• With linear models, usually more than one
  solution is D-optimal
• Orthogonality is means of reducing number of
  solutions
• Orthogonality also introduces desirable secondary
  properties
• Separates effects of input factors (avoids
  aliasing)
• Makes estimates for elements of ? uncorrelated
• Orthogonal designs are not generally D-optimal
  D-optimal designs are not generally
  orthogonal
• However, some designs are both
• Classical factorial (cubic) designs are
  orthogonal (and often D-optimal)

10
Example Orthogonal Designs, r 2 Factors
11
Example Orthogonal Designs, r 3 Factors
xk3
12
Response Surface Methodology (RSM)

• Suppose want to determine inputs x that minimize
  the mean response z of some process (E(z))
• There are also other (nonoptimization) uses for
  RSM
• RSM can be used to build local models with the
  aim of finding the optimal x
• Based on building a sequence of local models as
  one moves through factor (x) space
• Each response surface is typically a simple
  regression polynomial
• Experimental design can be used to determine
  input values for building response surfaces

13
Steps of RSM for Optimizing x

• Step 0 (Initialization) Initial guess at optimal
  value of x.
• Step 1 (Collect data) Collect responses z from
  several x values in neighborhood of current
  estimate of best x value (can use experimental
  design).
• Step 2 (Fit model) From the x, z pairs in step 1,
  fit regression model in region around current
  best estimate of optimal x.
• Step 3 (Identify steepest descent path) Based on
  response surface in step 2, estimate path of
  steepest descent in factor space.
• Step 4 (Follow steepest descent path) Perform
  series of experiments at x values along path of
  steepest descent until no additional improvement
  in z response is obtained. This x value
  represents new estimate of best vector of factor
  levels.
• Step 5 (Stop or return) Go to step 1 and repeat
  process until final best factor level is
  obtained.

14
Conceptual Illustration of RSM for Two Variables
in x Shows More Refined Experimental Design Near
Solution
Adapted from Montgomery (2001), Design and
Analysis of Experiments, Fig. 11-3
15
Nonlinear Design

• Assume model
• z h(?,?x) v ,
• where ? enters nonlinearly
• D-optimality remains dominant measure
• Maximization of determinant of Fisher information
  matrix (from Chapter 13 of ISSO Fn(?, x) is
  Fisher information matrix based on n data points)
• Fundamental distinction from linear case is that
  D-optimal criterion depends on ?
• Leads to conundrum
• Choosing x to best estimate ?, yet need to know
  ? to determine x

16
Strategies for Coping with Dependence on ?

• Assume nominal value of ? and develop an optimal
  design based on this fixed value
• Sequential design strategy based on an iterated
  design and model fitting process.
• Bayesian strategy where a prior distribution is
  assigned to ?, reflecting uncertainty in the
  knowledge of the true value of ?.

17
Sequential Approach for Parameter Estimation and
Optimal Design

•   Step 0 (Initialization) Make initial guess at
  ?, Allocate n0 measurements to initial
  design. Set k 0 and n 0.
• Step 1 (D-optimal maximization) Given Xn , choose
  the nk inputs in X to maximize
• Step 2 (Update ? estimate) Collect nk
  measurements based on inputs from step 1. Use
  measurements to update from to
• Step 3 (Stop or return) Stop if the value of ? in
  step 2 is satisfactory. Else return to step 1
  with the new k set to the former k 1 and the
  new n set to the former n nk (updated Xn now
  includes inputs from step 1).

18
Comments on Sequential Design

• Note two optimization problems being solved one
  for ?, one for ?
• Determine next nk input values (step 1)
  conditioned on current value of ?
• Each step analogous to nonlinear design with
  fixed (nominal) value of ?
• Full sequential mode (nk 1) updates ? based
  on each new input?ouput pair (xk , zk)
• Can use stochastic approximation to update ?
• where

19
Bayesian Design Strategy

• Assume prior distribution (density) for ?, p(?),
  reflecting uncertainty in the knowledge of the
  true value of ?.
• There exist multiple versions of D-optimal
  criterion
• One possible D-optimal criterion
• Above criterion related to Shannon information
• While log transform makes no difference with
  fixed ?, it does affect integral-based solution.
• To simplify integral, may be useful to choose
  discrete prior p(?)