Method of Least Squares

1
Method of Least Squares

• K Sudhakar, Amitay Isaacs, Devendra Ghate
• Centre for Aerospace Systems Design Engineering
• Department of Aerospace Engineering
• Indian Institute of Technology
• Mumbai 400 076

2
Method of Least Squares

• Response y can be created for any x
• Design ? a particular value of x vector
• Experiment generating Y for a design
• Let

3
Method of Least Squares

• Note
• Fit is linear in ?
• Polynomial terms for ? are natural choice
• suggested by Taylor series. ? will then be
• derivatives of f(x)

4
An Example
5
Method of Least Squares

• An experiment at design, gives yei
• N experiments are conducted, N gt k
• ? are mutually independent E?i?j ?ij?2
• Least Squares estimate of is

6
An Example
7
Method of Least Squares
8
Method of Least Squares

• How good is b as an estimate of ??
• Least Squares is an unbiased estimator

9
Method of Least Squares

• How good is b as an estimate of ??

10
Expectation, Variance of b?

• Estimate 1
• Choose N points Design
• Evaluate N responses
• Estimate b

• Estimate 2
• Evaluate N responses
• Estimate b again

11
Variance-Covariance Matrix of Vector b

• Variance-Covariance of b (XTX)-1 ?2
• ? is fixed for an experimental technique
• (XTX)-1 depends on where experiments are
  conducted, ie. xei
• Variance-Covariance of b can be reduced by
  suitable choice of xei i1, N
• Design Of Experiments - DOE

12
Predictive Capability

• Variance of predicted response depends on
• ?
• (XTX)-1 where experiments were conducted
• Point where prediction is being made

13
Role of Matrix X

• Variance-Covariance of b (XTX)-1 ?2
• Variance in prediction at a point P,
• (xei, i1, N) ? ? Design Of Experiments (DOE)
• DOE aims to make
• (XTX)-1 small?
• estimated parameters, b, un-correlated. (XTX)-1
  to be diagonal. Orthogonal design ? (XTX)-1
  diagonal
• designs rotatable? Error in predictions depend
  only on distance from center of design

14
DOE

• How to make (XTX)-1 small?
• (XTX)-1 is real, symmetric matrix
• (?i i1, k) are eigenvalues, of (XTX)-1
• Minimize
• tr (XTX)-1 ? ?i A-Optimality
• (XTX)-1 ? ?i D-Optimality
• max (?i) E-Optimality
• ?T (XTX)-1 ? G-Optimality
• D-Optimality implies G-Optimality

15
DOE

• Hypercube in ?n. xL ? x ? xU
• DOE points are available for
• first second order designs

16
An Example

• Yt 3 2x1 1x2 1 x1 x2
• Ye Yt ? ? RN(0, 0.2)
• -1 ? x1 ? 1 -1 ? x2 ? 1

17
Coded Variables
18
First Order Models Designs
Consider 2n design All corners of the
hypercube
19
Designs

• First Order Models
• 2k Factorial Design. 2 levels for each variable.
• Fractional replicates of 2k factorial design
• Simplex Design
• Placket-Burman Design
• Second Order Models
• 3k Factorial Design. 3 levels for each variable.
• Box-Behnken Design. Incomplete 3k factorial
• Central Composite Design.
• 2k Factorial Design points
• no centre points
• 2k axial points 2 points along each axis at a
  distance ?

20
Is the Fit Good?
Highly improbable Set of occurrences
21
Testing of Fit

• Variance in experiment, ?2?
• Known through careful assessment of experimental
  technique, sensors used, etc.
• Estimated experimentally. n repeat experiments
  at same xe
• Sum of Squares due to lack of fit
• SSLOF ? (yp - ye)2 /(N-1)
• F ?2 / SSLOF ? F Statistics
• Note If the fit closely passes through all
  points
• then F takes large value!

22
Tests of Hypothesis for ?i

• Null hypothesis, Ho ?i 0?
• Claim is that mean of ?i 0
• Fit has predicted mean as ?i ?ip
• Consider the t-statistic, t (?ip- 0)/??ip
• ??ip is available from XTX matrix
• Accept or reject hypothesis from t? at a suitable
  ? level

23
Analysis of Variance - ANOVA

• F Statistics
• t Statistics
• R2
• R2 Adjusted
• PRESS
• Various Tests to investigate fit
• To be discussed later

24
Generalized Least Squares/Maximum Likelihood

• Experimental errors ?i RN(0, ?i)
• E?i ?j 0

25
Generalized Least Squares/Maximum Likelihood

• Experimental errors ?i RN(0, ?i)
• E?i ?j 0

26
Generalized Least Squares/Maximum Likelihood
27
Generalized Least Squares/Maximum Likelihood
28
Generalized Least Squares/Maximum Likelihood