Multiple Regression

1
Multiple Regression

• Chapter 18

2
18.1 Introduction

• In this chapter we extend the simple linear
  regression model, and allow for any number of
  independent variables.
• We expect to build a model that fits the data
  better than the simple linear regression model.

3
Introduction

• We shall use computer printout to
• Assess the model
• How well it fits the data
• Is it useful
• Are any required conditions violated?
• Employ the model
• Interpreting the coefficients
• Predictions using the prediction equation
• Estimating the expected value of the dependent
  variable

4
18.2 Model and Required Conditions

• We allow for k independent variables to
  potentially be related to the dependent variable
• y b0 b1x1 b2x2 bkxk e

Coefficients
Random error variable
5
Multiple Regression for k 2, Graphical
Demonstration - I
y
The simple linear regression model allows for one
independent variable, x y b0 b1x e
y b0 b1x
y b0 b1x
y b0 b1x
y b0 b1x
Note how the straight line becomes a plain,
and...
y b0 b1x1 b2x2
y b0 b1x1 b2x2
y b0 b1x1 b2x2
y b0 b1x1 b2x2
y b0 b1x1 b2x2
X
y b0 b1x1 b2x2
1
y b0 b1x1 b2x2
The multiple linear regression model allows for
more than one independent variable. Y b0 b1x1
b2x2 e
X2
6
Multiple Regression for k 2, Graphical
Demonstration - II
y
y b0 b1x2
Note how a parabola becomes a parabolic Surface.
b0
X
1
y b0 b1x12 b2x2
X2
7
Required conditions for the error variable

• The error e is normally distributed.
• The mean is equal to zero and the standard
  deviation is constant (se) for all values of y.
• The errors are independent.

8
18.3 Estimating the Coefficients and
Assessing the Model

• The procedure used to perform regression
  analysis
• Obtain the model coefficients and statistics
  using a statistical software.

• Diagnose violations of required conditions. Try
  to remedy problems when identified.

• Assess the model fit using statistics obtained
  from the sample.

• If the model assessment indicates good fit to the
  data, use it to interpret the coefficients and
  generate predictions.

9
Estimating the Coefficients and Assessing the
Model, Example

• Example 18.1 Where to locate a new motor inn?
• La Quinta Motor Inns is planning an expansion.
• Management wishes to predict which sites are
  likely to be profitable.
• Several areas where predictors of profitability
  can be identified are
• Competition
• Market awareness
• Demand generators
• Demographics
• Physical quality

10
Margin
Profitability
Competition
Market awareness
Customers
Community
Rooms
Nearest
Office space
College enrollment
Income
Disttwn
Median household income.
Distance to downtown.
Distance to the nearest La Quinta inn.
Number of hotels/motels rooms within 3 miles
from the site.
11
Estimating the Coefficients and Assessing the
Model, Example
Operating Margin
Profitability
Market awareness
Competition
Customers
Community
Rooms
Nearest
Office space
College enrollment
Income
Disttwn
Distance to downtown.
Median household income.
Distance to the nearest La Quinta inn.
Number of hotels/motels rooms within 3 miles
from the site.
12
Estimating the Coefficients and Assessing the
Model, Example

• Data were collected from randomly selected 100
  inns that belong to La Quinta, and ran for the
  following suggested model
• Margin b0 b1Rooms b2Nearest b3Office
  b4College b5Income b6Disttwn

Xm18-01
13
Regression Analysis, Excel Output
This is the sample regression equation
(sometimes called the prediction equation)
Margin 38.14 - 0.0076Number 1.65Nearest
0.020Office Space 0.21Enrollment
0.41Income - 0.23Distance
14
Model Assessment

• The model is assessed using three tools
• The standard error of estimate
• The coefficient of determination
• The F-test of the analysis of variance
• The standard error of estimates participates in
  building the other tools.

15
Standard Error of Estimate

• The standard deviation of the error is estimated
  by the Standard Error of Estimate
• The magnitude of se is judged by comparing it to

16
Standard Error of Estimate

• From the printout, se 5.51
• Calculating the mean value of y we have
• It seems se is not particularly small.
• QuestionCan we conclude the model does not fit
  the data well?

17
Coefficient of Determination

• The definition is
• From the printout, R2 0.5251
• 52.51 of the variation in operating margin is
  explained by the six independent variables.
  47.49 remains unexplained.
• When adjusted for degrees of freedom, Adjusted
  R2 1-SSE/(n-k-1) / SS(Total)/(n-1)
• 49.44

18
Testing the Validity of the Model

• We pose the question
• Is there at least one independent variable
  linearly related to the dependent variable?
• To answer the question we test the hypothesis
• H0 b0 b1 b2 bk
• H1 At least one bi is not equal to zero.
• If at least one bi is not equal to zero, the
  model has some validity.

19
Testing the Validity of the La Quinta Inns
Regression Model

• The hypotheses are tested by an ANOVA procedure (
  the Excel output)

MSR/MSE
k nk1 n-1
SSR
MSRSSR/k
MSESSE/(n-k-1)
SSE
20
Testing the Validity of the La Quinta Inns
Regression Model

• Variation in y SSR SSE.
• Large F results from a large SSR. Then, much of
  the variation in y is explained by the regression
  model the model is useful, and thus, the null
  hypothesis should be rejected. Therefore, the
  rejection region is

Rejection region FgtFa,k,n-k-1
21
Testing the Validity of the La Quinta Inns
Regression Model
Conclusion There is sufficient evidence to
reject the null hypothesis in favor of the
alternative hypothesis. At least one of the bi
is not equal to zero. Thus, at least one
independent variable is linearly related to y.
This linear regression model is valid
Fa,k,n-k-1 F0.05,6,100-6-12.17 F 17.14 gt 2.17
Also, the p-value (Significance F)
0.0000 Reject the null hypothesis.
22
Interpreting the Coefficients

• b0 38.14. This is the intercept, the value of y
  when all the variables take the value zero.
  Since the data range of all the independent
  variables do not cover the value zero, do not
  interpret the intercept.
• b1 0.0076. In this model, for each additional
  room within 3 mile of the La Quinta inn, the
  operating margin decreases on average by .0076
  (assuming the other variables are held constant).

23
Interpreting the Coefficients

• b2 1.65. In this model, for each additional
  mile that the nearest competitor is to a La
  Quinta inn, the operating margin increases on
  average by 1.65 when the other variables are
  held constant.
• b3 0.020. For each additional 1000 sq-ft of
  office space, the operating margin will increase
  on average by .02 when the other variables are
  held constant.
• b4 0.21. For each additional thousand students
  the operating margin increases on average by .21
  when the other variables are held constant.
  

24
Interpreting the Coefficients

• b5 0.41. For additional 1000 increase in
  median household income, the operating margin
  increases on average by .41, when the other
  variables remain constant.
• b6 -0.23. For each additional mile to the
  downtown center, the operating margin decreases
  on average by .23 when the other variables are
  held constant.

25
Testing the Coefficients

• The hypothesis for each bi is
• Excel printout

H0 bi 0 H1 bi ¹ 0
d.f. n - k -1
26
Using the Linear Regression Equation

• The model can be used for making predictions by
• Producing prediction interval estimate for the
  particular value of y, for a given values of xi.
• Producing a confidence interval estimate for the
  expected value of y, for given values of xi.
• The model can be used to learn about
  relationships between the independent variables
  xi, and the dependent variable y, by interpreting
  the coefficients bi

27
La Quinta Inns, Predictions
Xm18-01

• Predict the average operating margin of an inn at
  a site with the following characteristics
• 3815 rooms within 3 miles,
• Closet competitor .9 miles away,
• 476,000 sq-ft of office space,
• 24,500 college students,
• 35,000 median household income,
• 11.2 miles distance to downtown center.

MARGIN 38.14 - 0.0076(3815) 1.65(.9)
0.020(476) 0.21(24.5)
0.41(35) - 0.23(11.2) 37.1
28
La Quinta Inns, Predictions

• Interval estimates by Excel (Data Analysis Plus)

It is predicted, with 95 confidence that the
operating margin will lie between 25.4 and
48.8. It is estimated the average operating
margin of all sites that fit this category falls
within 33 and 41.2. The average inn would not
be profitable (Less than 50).
29
Assessment and InterpretationMBA Program
Admission Policy

• The dean of a large university wants to raise the
  admission standards to the popular MBA program.
• She plans to develop a method that can predict an
  applicants performance in the program.
• She believes a students success can be predicted
  by
• Undergraduate GPA
• Graduate Management Admission Test (GMAT) score
• Number of years of work experience

30
MBA Program Admission Policy

• A randomly selected sample of students who
  completed the MBA was selected. (See MBA).
• Develop a plan to decide which applicant to admit.

31
MBA Program Admission Policy

• Solution
• The model to estimate isy b0 b1x1 b2x2
  b3x3ey MBA GPAx1 undergraduate GPA
  UnderGPAx2 GMAT score GMATx3 years of
  work experience Work
• The estimated modelMBA GPA b0 b1UnderGPA
  b2GMAT b3Work

32
MBA Program Admission Policy Model Diagnostics

• We estimate the regression model then we check

Normality of errors
33
MBA Program Admission Policy Model Diagnostics

• We estimate the regression model then we check

The variance of the error variable
34
MBA Program Admission Policy Model Diagnostics
35
MBA Program Admission Policy Model Assessment
36
18.4 Regression Diagnostics - II

• The conditions required for the model assessment
  to apply must be checked.
• Is the error variable normally distributed?
• Is the error variance constant?
• Are the errors independent?
• Can we identify outlier?
• Is multicolinearity (intercorrelation)a problem?

Draw a histogram of the residuals
Plot the residuals versus the time periods
37
Diagnostics Multicolinearity

• Example 18.2 Predicting house price (Xm18-02)
  
• A real estate agent believes that a house selling
  price can be predicted using the house size,
  number of bedrooms, and lot size.
• A random sample of 100 houses was drawn and data
  recorded.
• Analyze the relationship among the four variables

38
Diagnostics Multicolinearity

• The proposed model isPRICE b0 b1BEDROOMS
  b2H-SIZE b3LOTSIZE e

The model is valid, but no variable is
significantly related to the selling price ?!
39
Diagnostics Multicolinearity

• Multicolinearity is found to be a problem.

• Multicolinearity causes two kinds of
  difficulties
• The t statistics appear to be too small.
• The b coefficients cannot be interpreted as
  slopes.

40
Remedying Violations of the Required Conditions

• Nonnormality or heteroscedasticity can be
  remedied using transformations on the y variable.
• The transformations can improve the linear
  relationship between the dependent variable and
  the independent variables.
• Many computer software systems allow us to make
  the transformations easily.

41
Reducing Nonnormality by Transformations
Transformations, Example.

• A brief list of transformations
• y log y (for y gt 0)
• Use when the se increases with y, or
• Use when the error distribution is positively
  skewed
• y y2
• Use when the s2e is proportional to E(y), or
• Use when the error distribution is negatively
  skewed
• y y1/2 (for y gt 0)
• Use when the s2e is proportional to E(y)
• y 1/y
• Use when s2e increases significantly when y
  increases beyond some critical value.

42
Durbin - Watson TestAre the Errors
Autocorrelated?

• This test detects first order autocorrelation
  between consecutive residuals in a time series
• If autocorrelation exists the error variables are
  not independent

Residual at time i
43
Positive First Order Autocorrelation

Residuals



0
Time




Positive first order autocorrelation occurs when
consecutive residuals tend to be similar.
Then, the value of d is small (less than 2).
44
Negative First Order Autocorrelation
Residuals



0
Time




Negative first order autocorrelation occurs when
consecutive residuals tend to markedly differ.
Then, the value of d is large (greater than 2).
45
One tail test for Positive First Order
Autocorrelation

• If dltdL there is enough evidence to show that
  positive first-order correlation exists
• If dgtdU there is not enough evidence to show that
  positive first-order correlation exists
• If d is between dL and dU the test is
  inconclusive.

46
One Tail Test for Negative First Order
Autocorrelation

• If dgt4-dL, negative first order correlation
  exists
• If dlt4-dU, negative first order correlation does
  not exists
• if d falls between 4-dU and 4-dL the test is
  inconclusive.

47
Two-Tail Test for First Order Autocorrelation

• If dltdL or dgt4-dL first order autocorrelation
  exists
• If d falls between dL and dU or between 4-dU and
  4-dLthe test is inconclusive
• If d falls between dU and 4-dU there is no
  evidence for first order autocorrelation

48
Testing the Existence of Autocorrelation, Example

• Example 18.3 (Xm18-03)
• How does the weather affect the sales of lift
  tickets in a ski resort?
• Data of the past 20 years sales of tickets, along
  with the total snowfall and the average
  temperature during Christmas week in each year,
  was collected.
• The model hypothesized was
• TICKETSb0b1SNOWFALLb2TEMPERATUREe
• Regression analysis yielded the following
  results

49
The Regression Equation Assessment (I)
Xm18-03
The model seems to be very poor

• R-square0.1200
• It is not valid (Signif. F 0.3373)
• No variable is linearly related to Sales

50
Diagnostics The Error Distribution
The errors histogram
The errors may be normally distributed
51
Diagnostics Heteroscedasticity
52
Diagnostics First Order Autocorrelation
The errors are not independent!!
53
Diagnostics First Order Autocorrelation
Using the computer - Excel
Tools gt Data Analysis gt Regression (check the
residual option and then OK) Tools gt Data
Analysis Plus gt Durbin Watson Statistic gt
Highlight the range of the residuals from the
regression run gt OK
Test for positive first order auto-correlation n
20, k2. From the Durbin-Watson table we have
dL1.10, dU1.54. The statistic
d0.5931 Conclusion Because dltdL , there is
sufficient evidence to infer that positive first
order autocorrelation exists.
The residuals
54
The Modified Model Time Included
The modified regression model (Xm18-03mod) TICKET
Sb0 b1SNOWFALL b2TEMPERATURE b3TIMEe

• All the required conditions are met for this
  model.
• The fit of this model is high R2 0.7410.
• The model is valid. Significance F .0001.
• SNOWFALL and TIME are linearly related to
  ticket sales.
• TEMPERATURE is not linearly related to ticket
  sales.