3.3 Hypothesis Testing in Multiple Linear Regression

1
3.3 Hypothesis Testing in Multiple Linear
Regression

• Questions
• What is the overall adequacy of the model?
• Which specific regressors seem important?
• Assume the errors are independent and follow a
  normal distribution with mean 0 and variance ?2

2

• 3.3.1 Test for Significance of Regression
• Determine if there is a linear relationship
  between y and xj, j 1,2,,k.
• The hypotheses are
• H0 ß1 ß2 ßk 0
• H1 ßj? 0 for at least one j
• ANOVA
• SST SSR SSRes
• SSR/?2 ?2k, SSRes/?2 ?2n-k-1, and SSR and
  SSRes are independent

3

• Under H1, F0 follows F distribution with k and
  n-k-1 and a noncentrality parameter of

4

• ANOVA table

5
(No Transcript)
6

• Example 3.3 The Delivery Time Data

7

• R2 and Adjusted R2
• R2 always increase when a regressor is added to
  the model, regardless of the value of the
  contribution of that variable.
• An adjusted R2
• The adjusted R2 will only increase on adding a
  variable to the model if the addition of the
  variable reduces the residual mean squares.

8

• 3.3.2 Tests on Individual Regression Coefficients
• For the individual regression coefficient
• H0 ßj 0 v.s. H1 ßj ? 0
• Let Cjj be the j-th diagonal element of (XX)-1.
  The test statistic
• This is a partial or marginal test because any
  estimate of the regression coefficient depends on
  all of the other regression variables.
• This test is a test of contribution of xj given
  the other regressors in the model

9

• Example 3.4 The Delivery Time Data

10

• The subset of regressors

11

• For the full model, the regression sum of square
• Under the null hypothesis, the regression sum of
  squares for the reduce model
• The degree of freedom is p-r for the reduce
  model.
• The regression sum of square due to ß2 given ß1
• This is called the extra sum of squares due to ß2
  and the degree of freedom is p - (p - r) r
• The test statistic

12

• If ß2 ? 0, F0 follows a noncentral F distribution
  with
• Multicollinearity this test actually has no
  power!
• This test has maximal power when X1 and X2 are
  orthogonal to one another!
• Partial F test Given the regressors in X1,
  measure the contribution of the regressors in X2.

13

• Consider y ß0 ß1 x1 ß2 x2 ß3 x3 ?
• SSR(ß1 ß0 , ß2, ß3), SSR(ß2 ß0 , ß1, ß3)
  and SSR(ß3 ß0 , ß2, ß1) are signal-degree-of
  freedom sums of squares.
• SSR(ßj ß0 ,, ßj-1, ßj, ßk) the contribution
  of xj as if it were the last variable added to
  the model.
• This F test is equivalent to the t test.
• SST SSR(ß1 ,ß2, ß3ß0) SSRes
• SSR(ß1 ,ß2 , ß3ß0) SSR(ß1ß0) SSR(ß2ß1, ß0)
  SSR(ß3 ß1, ß2, ß0)

14

• Example 3.5 Delivery Time Data

15

• 3.3.3 Special Case of Orthogonal Columns in X
• Model y Xß ? X1ß1 X2ß2 ?
• Orthogonal X1X2 0
• Since the normal equation (XX)ß Xy,

16
(No Transcript)
17

• 3.3.4 Testing the General Linear Hypothesis
• Let T be an m ? p matrix, and rank(T) r
• Full model y Xß ?
• Reduced model y Z? ? , Z is an n ? (p-r)
  matrix and ? is a (p-r) ?1 vector. Then
• The difference SSH SSRes(RM) SSRes(FM) with
  r degree of freedom. SSH is called the sum of
  squares due to the hypothesis H0 Tß 0

18

• The test statistic

19
(No Transcript)
20

• Another form
• H0 Tß c v.s. H1 Tß? c Then

21
3.4 Confidence Intervals in Multiple Regression

• 3.4.1 Confidence Intervals on the Regression
  Coefficients
• Under the normality assumption,

22
(No Transcript)
23

• 3.4.2 Confidence Interval Estimation of the Mean
  Response
• A confidence interval on the mean response at a
  particular point.
• x0 (1,x01,,x0k)
• The unbiased estimator of E(yx0)

24

• Example 3.9 The Delivery Time Data

25

• 3.4.3 Simultaneous Confidence Intervals on
  Regression Coefficients
• An elliptically shaped region

26

• Example 3.10 The Rocket Propellant Data

27
(No Transcript)
28

• Another approach
• ? is chosen so that a specified probability that
  all intervals are correct is obtained.
• Bonferroni method ? ta/2p, n-p
• Scheffe S-method ?(2Fa,p, n-p )1/2
• Maximum modulus t procedure ? ua,p, n-2 is the
  upper ? tail point of the distribution of the
  maximum absolute value of two independent student
  t r.v.s each based on n-2 degree of freedom

29

• Example 3.11 The Rocket Propellant Data
• Find 90 joint C.I. for ß0 and ß1 by
  constructing a 95 C.I. for each parameter.

30

• The confidence ellipse is always a more efficient
  procedure than the Bonferroni method because the
  volume of the ellipse is always less than the
  volume of the space covere3d by the Bonferroni
  intervals.
• Bonferroni intervals are easier to construct.
• The length of C.I.
• Maximum modulus t lt Bonferroni method
• lt Scheffe S-method

31
3.5 Prediction of New Observations
32
3.6 Hidden Extrapolation in Multiple Regression

• Be careful about extrapolating beyond the region
  containing the original observations!
• Rectangle formed by ranges of regressors NOT data
  region.
• Regressor variable hull (RVH) the convex hull of
  the original n data points.
• Interpolation x0 ? RVH
• Extrapolation x0 ? RVH

33
(No Transcript)
34

• hii of the hat matrix H X(X?X)-1Xare useful in
  detecting hidden extrapolation.
• hmax the maximum of hii . The point xi that has
  the largest value of hii will lie on the boundary
  of RVH
• x x?(X?X)-1x ? hmax is an ellipsoid
  enclosing all points inside the RVH.
• Let h00 x0'(X'X)-1x0
• h00 ? hmax inside the RVH and the boundary of
  RVH
• h00 gt hmax outside the RVH

35

• MCE minimum covering ellipsoid (Weisberg,
  1985).

36
(No Transcript)
37
3.7 Standardized Regression Coefficients

• Difficult to compare regression coefficients
  directly.
• Unit Normal Scaling Standardize a Normal r.v.

38

• New model
• There is no intercept.
• The least-square estimator of b is

39

• Unit Length Scaling

40

• New Model
• The least-square estimator

41

• It does not matter which scaling we use! They
  both produce the same set of dimensionless
  regression coefficient.

42
(No Transcript)
43
(No Transcript)
44
3.8 Multicollinearity

• A serious problem Multicollinearity or
  near-linear dependence among the regression
  variables.
• The regressors are the columns of X. So an exact
  linear dependence would result a singular XX

45

• Unit length scaling

46

• Soft drink data
• Off-diagonal elements are of WW usually called
  the simple correlations between regressors.

47

• Variance inflation factors (VIFs)
• The main diagonal elements of the inverse of XX
  ((WW)-1 above)
• From above two casesSoft drink VIF1 VIF2
  3.12 and Figure 3.12 VIF1 VIF2 1
• VIFj 1/(1-Rj)
• Rj is the coefficient of multiple determination
  obtained from regressing xj on the other
  regressor variables.
• If xj is nearly linearly dependent on some of the
  other regressors, then Rj ? 1 and VIFj will be
  large.
• Serious problems VIFs gt 10

48

• Figure 3.13 (a) The plan is unstable and very
  sensitive to relatively small changes in the data
  points.
• Figure 3.13 (b) Orthogonal regressors.

49
3.9 Why Do Regression Coefficients Have the Wrong
Sign?

• The reasons of the wrong sign
• The range of some of the regressors is too small.
• Important regressors have not been included in
  the model.
• Multicollinearity is present.
• Computational errors have been made.

50

• For reason 1

51

• Although it is possible to decrease the variance
  of the regression coefficients by increase the
  range of the xs, it may not be desirable to
  spread the levels of the regressors out too far
• The true response function may be nonlinear.
• Impractical or impossible.
• For reason 2

52

53

• Fore reason 3 Multicollinearity inflates the
  variances of the coefficients, and this increases
  the probability that one or more regression
  coefficients will have the wrong sign.
• Different computer programs handle round-off or
  truncation problems in different ways, and some
  programs are more effective than the others in
  this regard.