Simulation Matching and Model Estimation Using Statistical Learning Techniques

1
Simulation Matching and Model Estimation Using
Statistical Learning Techniques
Principal investigator Prof. Nicholas
Zabaras Other participants Jingbo Wang and V.A.
Badri Narayanan Materials Process Design and
Control Laboratory Sibley School of Mechanical
and Aerospace Engineering188 Frank H. T. Rhodes
Hall Cornell University Ithaca, NY
14853-3801 Email zabaras_at_cornell.edu Phone
[phone] URL http//www.mae.cornell.edu/zab
aras/
Materials Process Design and Control Laboratory
2
What is the problem?

• For a given complex system, we have
• A collection of data from experiments
  (tests) conducted in the past
• Can this information help us to
• Estimate the system parameters with specified
  confidence intervals
• Predict the outcome of the experiment (test)
  given a new input
• Identify adjustment to the system parameters to
  meet customer performance requirements
• or even further
• Identify modifications in the experimentation
  simulation methods to improve matching their
  results

The answer is yes and it lies in statistical
machine learning methodologies
Materials Process Design and Control Laboratory
3
Proposed approach --- Statistical Machine Learning
Input data
System model (parameters)
Output prediction
Data from previous experiments

• In statistical machine learning
• Models (Learners) built using previous data
  (Training sets) enable us to estimate the system
  parameters and predict outcome of new unseen
  inputs
• Obtaining this information can help us in
  applications of design and control
• These methods have strong roots in statistics
  and employ highly effective database techniques

Materials Process Design and Control Laboratory
4
Statistical learning An introduction

• Learning from data

Inputs
Quantitative
Training set
Data (sample set)
Outputs
Qualitative
Testing set

• Model Learner Predictive model

Supervised learning (Presence of output in model)
Regression (Quantitative output)

• Learning Paradigms

Classification (Qualitative output)
Unsupervised learning (No output in model)
Materials Process Design and Control Laboratory
5
Statistical decision framework

• Loss function --- error between prediction and
  measurement

X ,q
))
(
,
(

f
Y
L
where,

Î
vector (measurement)
Random output

Rk
Y
X ,q
Functional relation between X and Y
)
(
f

• In a general framework, we try to minimize the
  expected loss from prediction
• Depending on the loss function used and the type
  of model (with known or unknown functional
  relations), we have different learning methods

Materials Process Design and Control Laboratory
6
Popular statistical learning methods
Nature of output data
Categorical (e.g. On, Off)
Quantitative

• Pattern recognition and associative methods
• Nearest neighbor methods
• Functional approximation techniques
• Maximum a posteriori probability method
• Neural networks

• Regression analysis
• Partial/Ridge/Lasso/ PCA and other subset
  techniques
• Bayesian estimates

• Discriminant analysis
• Regression on indicator sets
• ANOVA
• Maximum likelihood estimates

Regression
Common methods
Classification
These methods are few of the most popular
statistical learning methods available
Materials Process Design and Control Laboratory
7
Functional input-output relation not known

• This is a pure data mining problem. In this
  case,
• the solution sequence is as follows
• - study the distribution of data helps to
  visualize the spread of data, detect the presence
  of outliers, etc.
• - choose the basis functions to fit the data in
  case the input/output relationship is perceived
  to be nonlinear
• - use LS or other optimization methods to
  evaluate the weight associated with each basis
  function
• - Build the model this model is called as
  learner and is used for output prediction and
  parameter estimation

Materials Process Design and Control Laboratory
8
Functional input-output relation not known

• A predictive model can be designed using linear
  regression

Let the inputs be (X1,X2) and the output Y.
First assume that the output is a function of
(X1, X2, X1X2, ). Then a subset
selection may indicate that X1 X1X2 best
explain the data
2
2
X
X
,
1
2
No
Yes
Exit
Materials Process Design and Control Laboratory
9
Known functional input-output relation

• With known functional relationship between input
  and output, the problem is reduced to that of
  estimation of the function (system) parameters
  that best explain the data
• Here it is assumed that once the optimal values
  of system parameters are known, the output can be
  estimated from the input using the given
  functional relationship
• System parameter estimation is performed in
  steps
• Input data reduction and smoothing for large
  data set (Kernel
• methods, spline bases, etc.)
• Selection of type of approximation (Linear or
  nonlinear)
• Choice of model (Robust regression, Maximum
  likelihood, Bayesian
• and other PDF based methods, Neural
  networks, etc.)

Materials Process Design and Control Laboratory
10
Formulation of the problem by statistical learning
System parameters (?1,?2,..., ?s)
Input data (X1,X2,...,Xp)
Predicted output (Y1,Y2,...,Yk)

• For a complex system as shown above
• Xi, ?i, Yi --- all random variables
• The functional form of input output relation is
  known
• A sample data set is available
• and the following information
• Known statistical nature of input and output
• Known uncertainty in the system parameters
• can we
• Estimate optimal values for the system parameters
  that can allow accurate prediction of future test
  cases?
• Provide confidence intervals for the optimal
  system parameters?
• Predict future outcomes of the model with desired
  accuracy?

Materials Process Design and Control Laboratory
11
An example
Simple beam simulation matching problem
Materials Process Design and Control Laboratory
12
Example problem (schematic)
Consider the beam shown
L
P
a
x
h
E Elastic Modulus
y
b
Well known equations for the maximum deflection
and maximum stress in terms of the system
parameters
2


6
P
a



-

2
P
a
a
3
L
(
)

s
max

y
max
2
3

b
h


E
b
h
Materials Process Design and Control Laboratory
13
Schematic of the testing cycle
Uncertainty added due to manufacturing variations
Modified beam parameter values are uncertain
Compliance design
Prototype
To consider any design modifications for better
performance in customer operating conditions
Manufacturing process
Compliance testing
Pre-compliance design beam
Actual production beam parameters may vary from
those of compliance test
Currently produced beams

• Testing facility has to be recorrelated for
  every new beam

Test conditions variations, measurement errors
and other undefined uncertainties
Materials Process Design and Control Laboratory
14
Problem statement

• Simulation of the beam cycle matching problem
• A design specification (L, b, h, E) for the beam
  is introduced. These pre-compliance parameters
  serve as basis for production of the prototype
  beam.
• Since the operating conditions used for testing
  may not be the same as the customer operating
  conditions, the inputs are adjusted to give the
  same maximum operating stress.
• Prototype of the beam is produced based on
  pre-compliance specifications and compliance
  tests are carried out to consider any required
  design modifications to the beam that can improve
  its performance under customer operating
  conditions.
• Depending on the compliance tests the beam
  parameter values are updated. These values are
  used in subsequent production of beams.
• Beams are now produced. Production acceptance
  tests are carried out on each of the manufactured
  beams. These tests are carried out in the
  presence of the customer under test operating
  conditions that best represent the customer
  operating conditions.

Materials Process Design and Control Laboratory
15
Problem statement

• Uncertainties are introduced in the above
  framework due to following reasons
• Changes in manufacturing characteristics (L, b,
  h, E) after compliance test
• Manufacturing variations Variations introduced
  during production due to tooling errors,
  machining allowances, etc.
• Test variations Variations in test site
  surrounding conditions for e.g. temperature,
  humidity, presence of any new structure, error in
  measurement of test quantities, etc.
• Other measurement errors Errors introduced in
  final reported values of beam parameters due to
  measurement uncertainty
• The beams in production can show different
  results if subjected to compliance tests

Materials Process Design and Control Laboratory
16
Objectives and questions to address
Data collected from previous production
acceptance tests and compliance tests
Can we estimate the beam parameters along with
their intervals of confidence?
Can we update the model as and when new test data
arrive instead of building the model over again?
Materials Process Design and Control Laboratory
17
Problem statement ...

• Information provided
• Pre-compliance design values for the beam
  parameters
• A model for the beam that provides functional
  input-output relationship. Inputs here are the
  load (P) and the point of application (a). The
  measured outputs are the maximum stress (smax)
  and the maximum deflection (Ymax)
• A brief description of compliance and acceptance
  test conditions
• Estimates of the manufacturing and measurement
  variations (actual values are not known). These
  estimates would be of use in probabilistic
  methods like Bayesian methods
• Data collected based on 20 acceptance tests and 2
  compliance tests. This data comprises of the test
  inputs and the measured outputs

Materials Process Design and Control Laboratory
18
Objectives

• Based on the provided information
• Obtain optimal estimates for the beam parameters
  (L, b, h, E) such that future use cases can be
  predicted with best possible accuracy
• Compute the intervals of confidence for these
  estimates
• Predict the test outcomes for new unseen inputs,
  perform accuracy studies for the above obtained
  beam parameter values test error,
  cross-validation, etc.
• Predict and/or explain what would be the outcome
  if compliance tests were performed on the beams
  in production.
• Investigate the effect of manufacturing and
  measurement variations on the estimated
  parameters This can be viewed as a sensitivity
  problem

Several methods can be used to achieve these
objectives
Materials Process Design and Control Laboratory
19
Three individual methods were examined
Regression approach

• Mean squared loss
• Weighted mean square loss
• Linear robust loss

Materials Process Design and Control Laboratory
20
Software environment (S-Plus)
A typical S-Plus working environment
objects
analysis plots
data
Materials Process Design and Control Laboratory
21
General regression models

• In general regression theory, the output Y is
  modeled as a function of the input (X) and the
  system parameters (?) as follows

• A loss function that is a function of the output
  (Y), input (X) and the system parameters (?) is
  associated with the regression model. This
  regression problem is in general a hard problem

• In special cases where the output may be written
  as follows

(?1,...,?s),

(X1,...,XP)
f
Y
ß
m
m
we can use highly efficient variants of linear
regression (squared loss, weighted square loss
and robust regression) to find the optimal
regression parameters ßs and then use them for
prediction.
Materials Process Design and Control Laboratory
22
Forms of loss functions

• The type of regression methods vary with the
  form of loss functions used. Three of the most
  popular methods use the following loss functions

2
?
-

))
,
(
(
X
f
Y
E
loss
Squared
?
-
)
,
(
X
f
Y
r
)
(

E
loss
Robust
ˆ
s
?
-
S
?
-
-
1
T
)
,
(
(
))
,
(
(
X
f
Y
X
f
Y
E
loss
Weighted
f(X,?) is the known functional relation, S is the
covariance matrix of input random vector, ? is a
bounded loss function and s is a robust scale
parameter

• For the beam problem, we can express ymax and
  smax as follows

2
3


b
b
Pa
Pa
y
2
1
max

b
Pa
s
3
max
-
6
2
6L



b
b
b
,
,
3
2
1
2
3
3
bh
Ebh
Ebh
This type of input/output functional
relationships enable usage of linear regression
techniques for prediction purposes
Materials Process Design and Control Laboratory
23
System parameter estimation

• For sufficiently smooth loss functions, we can
  obtain the system parameters (?) by minimizing
  directly L with respect to the system parameters
  (using e.g. gradient optimization techniques).
  This is the case with squared loss and weighted
  squared loss regression.

• For a loss function with discontinuous
  gradients, we have to use the regression
  coefficients b to obtain estimates of the system
  parameters. This is the case for robust
  regression.

Materials Process Design and Control Laboratory
24
A gradient-based optimization approach
Direct approach (squared and weighted loss only)

• We minimize the loss function L directly with
  respect to the system parameters ? (L, b, h,
  E), thus obtaining optimal estimates. System
  parameter evaluation here does not require
  knowledge of the regression parameters b.

• The loss function is expanded in a Taylor series
  about each design point as follows

• For positive definite Hessian of L, we obtain a
  unique step size Dqk

Materials Process Design and Control Laboratory
25
A gradient based optimization approach ...

• Mean squared loss function

loss
Squared
-
-
)2
ˆ
(
Deflection
)2,
ˆ
(
Stress
ymax
ymax
E
smax
smax
E
ˆ
smax and ymax are the predictions of the
maximum stress and deflection Correlation
between outputs is ignored leading to different
parameter estimates for stress and deflection

• The two outputs can be correlated by using the
  weighted loss function with Mahalonobis distance

ˆ
ˆ
-
S
-
-
T
1
Y
Y
Y
E
)
(
(
loss
Weighted
Y
)
ˆ


ymax
Y

and

where
T
Y
T
,
smax
,
ˆ
smax
ˆ
ymax
Materials Process Design and Control Laboratory
26
Estimation of parameters for robust regression

• The loss function of robust regression is given
  as
• ?(.) is symmetric bounded loss function and
• s is a robust scale estimate for residuals

Note S-plus allows for various functional forms
of ?(.) and s.

• For linear robust regression f(X,?) ßTX and the
  robust loss function is minimized with respect to
  the regression parameters b.
• The best estimate of ß is based on nearness to
  an appropriate linear regression estimate ßo
  which is defined by

• This is called a robust MM-estimator

Ref Yohai, V. J. and Zamar, R. H. (1998),
Optimal Locally Robust M-estimates of
Regression. Journal of Statistical Planning and
Inference, 64, 309-323.
Materials Process Design and Control Laboratory
As post-processing, after obtaining the
regression parameters, the system parameter
?L,b, h, E can be computed by solving the
following optimization problem
Above robust loss function has discontinuous
gradients, we use an indirect approach
27
Estimation of parameters for robust regression ...

• Since the robust loss function has discontinuous
  gradients, we use an indirect approach

• The regression parameters b are estimated using
  inbuilt programs in S-Plus. These parameters can
  be used for prediction.

• As post-processing and after obtaining b, the
  system parameters ?L, b, h, E can be computed
  by solving the following optimization problem

• This kind of prediction of system parameters
  does not always give good results due to biasing
  of regression coefficients b and the nature of
  the spread of data (which was the case for the
  beam problem)

• Prediction using robust regression is not
  affected by system parameter evaluation!

Materials Process Design and Control Laboratory
28
Advantages and disadvantages of robust regression

• Advantages
• Low error rates
• No assumptions are made about the statistical
  nature of error
• Prediction does not require information about the
  system parameters (less function evaluations)
• Outliers (e.g. erroneous or redundant data) can
  be identified and avoided for predictions
• What are Outliers, What problems do they cause ?
• Outliers are data from unusual events, for
  example measurement of deformation with an
  improperly mounted strain-gauge

Outliers do not follow the general trend in data
Outliers
Materials Process Design and Control Laboratory
29
Outliers ...

• Outliers are data that are too noisy due to bad
  data handling
• Outliers are redundant data i.e. data that
  convey nearly the same information
• Problems ...
• Parameter estimates depend on outlier data
• Probability density function of outliers are
  nearly the same as those of correct data near the
  center but tail probabilities differ considerably
• Increase Bias of the estimates or predictions
• Lead to inflated residual sum of squares

Inflated tail probabilities indicating presence
of outliers
PDF plot for smax
Disadvantage Robust regression may lead to
erroneous estimates for beam parameters if the
amount of data is small, however it can be used
to provide reliable predictions.
Materials Process Design and Control Laboratory
30
Results of the three regression methods

• The optimal system parameters and their
  Cross-Validation (CV) errors in prediction for
  the training set (provided by GE-AE) are
  tabulated below.
• Initial guess used was L, b, h, E 22.15in,
  2.08in, 1.092in, 0.995e07 -- these are the
  sample means for the production beams. The
  algorithm however showed high sensitivity to
  initial guess, thus sample mean was chosen as the
  best guess.
• The compliance test results were incorporated
  into the regression process with an importance
  factor Wimp between 0,1 (results shown here
  for Wimp0.1).

A question --- What is Cross-Validation?
Materials Process Design and Control Laboratory
31
Cross-Validation (CV)

• In general

Training set ( build model)
Sample data set
Testing set ( check performance of model)

• In the beam simulation problem, only training
  data is available (small sample data set). Then
  CV is used to test the general prediction error
  of the model.

• Ideas of Cross-Validation
• Total K data pairs (Xi, Yi), i 1, 2, , K
• Let (Xi, Yi) be the ith record

y
x
Materials Process Design and Control Laboratory
32
Cross-Validation (CV)

• In general

Training set ( build model)
sample data set
Testing set ( check performance of model)

• In the beam simulation problem, only training
  data is available (small sample data set). Then
  CV is used to test the general prediction error
  of the model

• Ideas of Cross-Validation
• Total K data pairs (Xi, Yi), i 1, 2, , K
• Let (Xi, Yi) be the ith record
• Temporarily remove (Xi, Yi) from the dataset

y
Materials Process Design and Control Laboratory
33
Cross-Validation (CV)

• In general

Training set ( build model)
sample data set
Testing set ( check performance of model)

• In the beam simulation problem, only training
  data is available (small sample data set). Then
  CV is used to test the general prediction error
  of the model

• Ideas of Cross-Validation
• Total K data pairs (Xi, Yi), i 1, 2, , K
• Let (Xi, Yi) be the ith record
• Temporarily remove (Xi, Yi) from the dataset
• Train on the remaining K-1 data points

y
x
Materials Process Design and Control Laboratory
34
Cross-Validation (CV)

• In general

Training set ( build model)
sample data set
Testing set ( check performance of model)

• In the beam simulation problem, only training
  data is available (small sample data set). Then
  CV is used to test the general prediction error
  of the model

• Ideas of Cross-Validation
• Total K data pairs (Xi, Yi), i 1, 2, , K
• Let (Xi, Yi) be the ith record
• Temporarily remove (Xi, Yi) from the dataset
• Train on the remaining K-1 data points
• Note your prediction error at (Xi, Yi)
• When youve done all points, report the
• mean error.

In the beam problem, we reported the relative CV
errors on the training data set
Materials Process Design and Control Laboratory
35
Measured maximum stress v.s. the prediction by
linear robust regression
4600
Estimate smax
4500
4400
4300
4200
4300
4400
4500
4600
4700
4800
smax
Materials Process Design and Control Laboratory
36
Measured maximum deflection v.s. the prediction
by linear robust regression
-0.038
-0.039
Estimate Ymax
-0.040
-0.041
-0.050
-0.045
-0.040
-0.035
-0.030
Ymax
Materials Process Design and Control Laboratory
37
Discussion of results from regression methods

• The linear robust regression has the minimum CV
  error but estimates of parameters are highly
  biased. It requires more combined parameters ß in
  order to get better estimates
• Square loss and weighted square loss gave almost
  the same estimates and CV errors (i.e. the two
  output quantities are highly independent on each
  other, which is also verified by the figure in
  the next page)
• Compliance test data were
• incorporated in the training set
• with different Wimps. CV error
• goes up with increasing Wimp
• Linear robust regression is the
• best in prediction.

Materials Process Design and Control Laboratory
38
Plot of dependencies in data
4.57
4.58
4.59
4.60
4.61
4.62
4100
4300
4500
4700
4900
22.00
22.05
22.10
22.15
22.20
22.25
1.000
1.025
1.050
1.075
1.100
1.125
20
15
10
description
5
0
22.25
22.20
L
22.15
22.10
22.05
2.10
22.00
2.08
2.06
b
2.04
2.02
2.00
1.125
1.100
h
1.075
1.050
1.025
420
1.000
410
P
400
390
380
4.62
4.61
a
4.60
4.59
4.58
4.57
-0.031
-0.036
ymax
-0.041
-0.046
-0.051
4900
4700
smax
4500
4300
4100
380
390
400
410
420
-0.051
-0.046
-0.041
-0.036
-0.031
0
5
10
15
20
2.00
2.02
2.04
2.06
2.08
2.10
The measured data does not seem to indicate any
possible dependencies
Materials Process Design and Control Laboratory
39
Bayesian approach
Maximum A Posteriori Probability Method
Materials Process Design and Control Laboratory
40
An introduction to a Bayesian approach (MAP)

• Whats new?
• - Know the uncertainty information (pdf) of
  parameters to be estimated
• - Start from a prior (known) pdf of parameters
  ( ) , derive a posterior
• pdf of parameters ( )
  conditioned on the given experiment data
• - Maximize the posterior pdf with respect to the
  parameters, i.e. find the
• parameters having the highest probability
  given the experimental data
• This is the so called Maximum A
  Posteriori Probability (MAP) Method

Bayesian approach - Makes use of Bayes
theorem
Materials Process Design and Control Laboratory
41
MAP Estimator of the example beam problem
Methodology Maximize
with respect to ? and XK, i.e. what are the most
possible ? and XK given YK
where, ? L, b, h, E ----- the system
parameter vector XK P(1),
a(1), P(2), a(2), , P(20),a(20) ----- input
vector YK ymax(1), smax(1),
ymax(2), smax(2), , ymax(20), smax(20) -----
output vector Loss function L -log(
P(?,XK YK) ) Minimize L with respect to ? and
XK. The denominator of P(?,XK YK) is taken as
constant in the optimization process. How to
obtain P(?), P(XK) and P( YK ?, XK )?
q
,
(
P
)

YK
XK
)
(
)
(
)
,

(
)
,
(
)
,

(
XK
P
P
XK
YK
P
XK
P
XK
YK
P
q
q
q
q


)

,
(
YK
XK
P
q
)
(
)
(
YK
P
YK
P
Materials Process Design and Control Laboratory
42
MAP Estimator of the example beam problem
Assumptions All manufacturing and testing
variations are independent Gaussian random noises
with zero mean. ?
X Functional relations (for each
single test) where, W
and We can obtain Y?,X
)
,
(
P
N
q
q
6
X
X
-
)
3
(
2
?
X
X
X

,
1
2
f
1

1
1
2
f
2
2
?
?
1
3
?
?
?
3
2
3
2
4
ù
é
f
1
)
,
(
P
N
ú
ê
W
f
û
ë
2
Materials Process Design and Control Laboratory
43
MAP Estimator of the example beam problem
Formulation of loss function where C is a
constant representing the denominator term of
P(?, XKYK ) Optimization of the loss function
(Gauss-Newton Method)
1
1

Materials Process Design and Control Laboratory
44
Evaluation of MAP estimator

• Biased and unbiased estimator
• - Let be the estimate of a unknown parameter,
  if
• E ( ) ?true
• Then, the estimator is unbiased. Otherwise, it
  is biased.
• - Unbiased estimator is desired because it
  reflects the true system parameter
• - A practical approach to check unbiasedness
  Monte-Carlo simulation
• Standard deviation of the estimates should be
  close to the
• true std values if known (beam problem)

Materials Process Design and Control Laboratory
45
Evaluation of MAP estimator by Monte-Carlo
Simulation
This function uses the arbitrary picked ?true to
generate outputs at a set of input points by
using the truth model of the system, namely the
f1 and f2 functions. Then it adds randomly
obtained noise to these outputs to generate
simulated measurements for the MAP estimator
Set iteration number N of M-C simulation and
?true
Truth model function of the beam
It returns an estimated parameter vector . If
the estimator is unbiased, expectation of
should be ?true
This averaged error should converge to zero for
large N
MAP estimator
Compute the error of estimation
Average the estimation error over N
Yes
No
Nth iteration?
A 1000 run of M-C simulation showed E( ) is a
good approximation of ?true
Materials Process Design and Control Laboratory
46
Results of MAP

• Data used in the MAP estimator
• The manufacturing variations were used in P?
• Measurement variations were used in PX and PW
• Mean of input vector took the value of the
  measurements
• All 20 acceptance testing data were used as
  training data set
• Two different means were used for ? the sample
  mean of 20 products and the pre-compliance
  specifications
• Computed results

Materials Process Design and Control Laboratory
47
Results of MAP
These errors were obtained by comparing prediction
s at compliance test condition with previous
compliance test results. It will be much more
valuable to compare with the compliance
test results done to production beams since the
design specifications have been changed
Materials Process Design and Control Laboratory
48
Discussion of results from MAP

• The standard deviations (confidence intervals) of
  parameter estimates computed by using product
  sample mean are much less than that by using
  pre-compliance specifications. The standard
  deviations of the former are very close to the
  true values.
• By using product sample mean, we also have less
  CV errors.
• Since the MAP estimator is unbiased and the
  estimates computed by using product sample mean
  have very accurate standard deviations, it is
  able to say it provides a precise estimation of
  the parameters.
• The MAP estimator is sensitive to the assumptions
  of manufacturing and measurement variations.
• Since the design parameters have been changed
  after compliance test, it makes no sense to use
  compliance test data in estimation. To improve
  the accuracy of estimation and prediction at
  compliance test condition, the modified design
  parameters (after compliance test) should be
  given as the mean of prior pdf of ?.

Materials Process Design and Control Laboratory
49
An extension of the MAP method

• A question ----- the manufacturing variation may
  not be
• available in a real situation
• Possible solution (pdf filtering)
• - Assume arbitrary variations for the parameters
  (but still Gaussian)
• - Use the product data (sample data) to compute
  a sample mean and sample variation
• - Start the estimation by using computed sample
  pdf
• - Update the prior pdf mean with new estimates
  and compute the new sample variations
• - Repeat the process until it converges
• With large set of data from product acceptance
  tests distributed over the
• full range of true pdf, the above algorithm is
  able to compute the best
• parameter without knowing the manufacturing
  variation

Materials Process Design and Control Laboratory
50
Illustration of Pdf filtering process
Probability
Value of ?
Materials Process Design and Control Laboratory
51
Comparison of the methods

• In general, the regression methods provide lower
  cross-validation errors
• Linear robust regression has lowest CV error in
  prediction of maximum
• deflection but the estimates are biased due to
  the nature of this method
• The Bayesian method has higher CV errors but it
  provides better estimates
• of the parameters (unbiased and small standard
  deviation)

Materials Process Design and Control Laboratory
52
Summary

• One can use a combination of linear robust
  regression and Bayesian
• method to perform the beam simulation and
  matching problem. Linear
• robust regression is used to predict the maximum
  deflection and stress at
• given new testing points and Bayesian method is
  used to estimate the
• system parameters and their confidence intervals.
• It will be helpful if
• Modified design parameters are known after
  compliance test
• The acceptance tests are performed at various
  points (different P and a)

Materials Process Design and Control Laboratory
53
Discussion of simulation matching process

• Key issues in simulation matching problem

Dominant output of system at compliance test
condition (smax in beam problem)
Nominal value of system output (requested by user)
Materials Process Design and Control Laboratory
54
Discussion of simulation matching process
Key issues in simulation matching problem
Dominant output of system at compliance test
condition (smax in beam problem)
Nominal value of system output (requested by user)
match
match
Prediction of system output at compliance test
(based upon best estimation of system
parameters)
Materials Process Design and Control Laboratory
55
Discussion of simulation matching process
Key issues in simulation matching problem
Dominant output of system at compliance test
condition (smax in beam problem)
Nominal value of system output (requested by user)
First stage
match
match
Prediction of system output at compliance test
(based upon best estimation of system
parameters)
- Compliance test is done for
prototypes of product - Modify the design
system parameter to meet nominal output -
Manufacturing products using modified
parameter - Repeat until nominal
value obtained
Materials Process Design and Control Laboratory
56
Discussion of simulation matching process
Key issues in simulation matching problem
Dominant output of system at compliance test
condition (smax in beam problem)
Nominal value of system output (requested by user)
First stage
Second stage
match
match
Prediction of system output at compliance test
(based upon best estimation of system
parameters)
- Compliance test is done for
prototypes of product - Modify the design
system parameter to meet nominal output -
Manufacturing products using modified
parameter - Repeat until nominal
value obtained

• - Estimation is based
• on data from product acceptance test
• More data, better accuracy
• The estimation process should be recursive (for
  each new data set,
• only need to update the
• previous estimate)

Materials Process Design and Control Laboratory
57
Discussion of simulation matching process
Key issues in simulation matching problem
Dominant output of system at compliance test
condition (smax in beam problem)
Nominal value of system output (requested by user)
First stage
Second stage
match
match
Prediction of system output at compliance test
(based upon best estimation of system
parameters)
- Compliance test is done for
prototypes of product - Modify the design
system parameter to meet nominal output -
Manufacturing products using modified
parameter - Repeat until nominal
value obtained

• - Estimation is based
• on data from product acceptance test
• More data, better accuracy
• The estimation process should be recursive (for
  each new data set,
• only need to update the
• previous estimate)

Materials Process Design and Control Laboratory
58
Need and objective of cycle matching
Objective To make accurate predictions on new
data, thereby avoiding expensive testing
Very expensive testing
Back to back testing
Nozzle testing
Bird hit testing
. . .
Information about all previously conducted tests
and simulations
Stress testing
Updating machine learning method
Statistical model learnt from a huge collection
of previous experiments
testing/simulation data for a new engine /engine
for maintenance
Update system parameters based on new test and
simulation results
Fluid flow interactions
Thermodynamics
. . .
Combustion
Highly complicated simulations
Materials Process Design and Control Laboratory
59
Recursive machine learning system for cycle
matching
A fully converged learnt system
Updated values for system parameters
Info
Full fledged system model based on data
Materials Process Design and Control Laboratory
60
Acknowledgements

• This preliminary work on statistical machine
  learning
• techniques and the beam cycle problem to test
  simulation matching techniques were supported by
  GE-GRC, Advanced Mechanical Technologies Program,
  Dr. R. Irani (program manager).
• Special thanks to Dr. R. Sampath (GE-GRC) for
  several clarifications on the problems examined.

Materials Process Design and Control Laboratory