Multiple Regression Analysis

1
Chapter 14

• Multiple Regression Analysis

2
Multiple Regression Analysis

• The general objective of regression analysis To
  model the relationship between a dependent
  variable y and one or more independent variables.
• For example, the variation of house prices can be
  attributed to house size, location, builders
  reputation, and other variables.
• Multiple regression model includes at least two
  predictor variables.
• Many of the concepts of simple linear regression
  can carry over to multiple regression with little
  modification.
• The calculations are more tedious, so a computer
  is an indispensable tool for multiple regression
  analysis.

3
14.1 Multiple Regression Models

• Deterministic model the value of y is completely
  determined once values of the independent
  variable have been specified.
• y a ß1x1 ß2x2 ßkxk
• Example In a school district, y 38000 800x1
  60x2 indicates that a teacher starts at salary
  ( y) of 38,000, and receives an additional 800
  per year for each year of teaching experience
  (x1) and 60 per year for each unit of post
  college coursework (x2) .
• The value of y is entirely determined by x1 and
  x2 through the deterministic formula. If two
  different teachers both have same number of years
  teaching experience (x1) and same number of post
  college units (x2) , they will have the same
  salary (y).

4
A general additive multiple regression models

• A probabilistic model is more realistic in most
  situations
• y a ß1x1 ß2x2 ßkxk e
• The random deviation e is assumed to be normally
  distributed with mean value 0 and standard
  deviation s.
• This implies that for fixed x1 , x2 , xk
  values, y has a normal distribution with standard
  deviation s and
• mean y value a ß1x1 ß2x2 ßkxk
• The ßis is called population regression
  coefficients
• The deterministic portion a ß1x1 ß2x2
  ßkxk is called the population regression
  function.

5
Multiple Regression Models

• Example What factors contribute to the academic
  success of college sophomores? Data collected in
  a survey of 1000 sophomores suggests that GPA at
  the end of second year is related to the
  students level of interaction with faculty and
  staff, and to the students commitment to his/her
  major.
• y GPA at the end of the second year.
• x1 level of faculty and staff interaction (a
  scale from 1 to 5)
• x2 level of commitment to major (a scale from
  1 to 5)
• One possible population model might be a 1.4,
  ß10.33 and ß2 0.16 y 1.4 .33x1 0.16x2
  with s 0.15.
• Find mean GPA for students whose x14.2 and
  x22.1.
• An actual y value will be within 2s of the mean
  value.

6
A Special Case Polynomial Regression

• Suppose that a scatterplot of the n sample (x, y)
  pairs has one of the appearances in the figure. A
  polynomial regression model is clearly more
  appropriate.

7
kth-Degree Polynomial Regression Model

• y a ß1x ß2x2 ßkxk e is a special
  case of the general multiple regression model
• y a ß1x1 ß2x2 ßkxk e
• with x1 x, x2x2, x3x3, xkxk.
• The most important special case other than simple
  linear regression (k 1) is the quadratic
  regression model.
• y a ß1x ß2x2 e
• A less encountered special case is that of cubic
  regression
• y a ß1x ß2x2 ß3x3 e

8
Interaction Between Variables

• y product from a certain chemical reaction
• x1 reaction temperature,
• x2 reaction pressure
• A chemist suggests for 80 x1 100 and 50 x2
  70 the probabilistic model
• y 1200 15 x1 35 x2 e.
• x1 90 mean y value 1200 15(90) 35 x2
  2550 35 x2
• x1 95 mean y value 1200 15(95) 35 x2
  2625 35 x2
• x1 100 mean y value 1200 15(100) 35 x2
  2700 35 x2
• Each graph is a straight line with the same slope
  35

9
Interaction Between Variables

• The three lines indicates that the average change
  in yield when pressure x2 is increased by 1 units
  is 35, regardless of temperature.
• But chemical theory suggests that the decline in
  average yield when pressure x2 increases should
  be more rapid for a high temperature than for a
  low temperature.

10
Interaction Between Variables

• Therefore, the line for a temperature x1 100
  should have a steeper decline than the line x1
  95, and that line in turn should be steeper than
  the one for x1 90.
• The previous model may not be appropriate.
• A model with this property included can be
• y 4500 75 x1 60x2 x1 x2 e
• x1 90 y 2250 30 x2
• x1 95 y 2625 35 x2
• x1 100 y 3000 40 x2

11
Interaction Between Variables

• If the change in the mean y value associated with
  a 1-unit increase in one variable depends on the
  value of a second variable, there is interaction
  between these two variables. When the variables
  are denoted by x1 and x2, such interaction can be
  modeled by including x1 x2, the product of the
  variables that interact, as a predictor variable.
• The multiple regression model based on two
  independent variables x1 and x2 and includes an
  interaction predictor is
• y a ß1x1 ß2x2 ß3 x1 x2 e.
• If there are 3 independent variables, a possible
  model is
• y a ß1x1 ß2x2 ß3 x3 ß4x4 ß5x5 ß6
  x6 e,
• where x4 x1 x2. x5 x1 x3 , x6 x2 x3 .

12
Nonlinear Relationship and Transformations

• Often a scatterplot exhibits a curved pattern
  indicating a nonlinear relationship between y and
  one of the variables xi.
• One method to find a curve to fit the data is to
  find a way to transform the xi value.
• A transformation involves using a simple function
  of a variable in place of the variable itself.
  For examples

13
Nonlinear Multiple Regression Models

• A frequently used model involving two independent
  variables x1 and x2 but k 5 predictors is the
  full quadratic or complete second-order model
• y a ß1x1 ß2x2 ß3 x1x2 ß4x12 ß5x22
  e
• When x1 has a fixed value, the graph of the
  regression function for x2 is a parabola.
• Many nonlinear relationship can be put into the
  form
• y a ß1x1 ß2x2 ßk xk e by
  transforming one or more of the variables. An
  appropriate transformation could be suggested by
  theory or by various plots of data.
• There are also relationships that cannot be
  linearized by transformation, and more
  complicated methods of analysis must be used.

14
Example Wind Chill Factor

• The wind chill index combines information on air
  temperature and wind speed to describe how cold
  it really feels. The following table gives the
  wind chill index for various combination of air
  temperature and wind speed.

15
Example Wind Chill Factor

• y wind chill index
• x1 air temperature,
• x2 wind speed
• Observation The wind chill index y increases
  linearly with air temperature x1 at each value of
  the wind speeds x2, but the linear pattern for
  the different wind speeds are not parallel.
• An interaction is appropriate.

16
Example Wind Chill Factor

• Observation The relationship between wind chill
  index and wind speed is nonlinear at each of the
  different temperatures.
• The pattern is more markedly curved at some
  temperatures than at others.
• Therefore a transformation is necessary.

17
Example Wind Chill Factor

• y wind chill index
• x1 air temperature, x2 wind speed
• The model used by the National Weather Service
  for relating wind chill index to air temperature
  and wind speed is
• mean y value 35.74 0.621 x1 35.75x2'
  0.4275 x1 x2',
• where x2' x20.16
• This model incorporates a transformed x2 to model
  the nonlinear relationship between wind chill
  index and wind speed, and an interaction term.

18
Example Predictors of Writing Competence

• The article Grade Level and Gender Differences
  in Writing Self-Beliefs of Middle School
  Students considered relating writing competence
  score to a number of predictor variables,
  including perceived value of writing and gender.
• Let y writing competence score
• x1 gender (x1 0 if male, and x1 1 if
  female)
• x2 perceived value of writing
• One possible multiple regression model is
• y a ß1x1 ß2x2 e
• Another possible model with an interaction term
  is
• y a ß1x1 ß2x2 ß3 x1 x2 e

19
Example Predictors of Writing Competence(no
interaction)

• For multiple regression model
• y a ß1x1 ß2x2 e
• when 0 (male)
• average y score a ß2x2 ,
• when 1 (female)
• average y score a ß1 ß2x2 ,
• where ß1 is the coefficient is the difference in
  average writing score between male and female
  when is x2 fixed.
• The two lines are parallel with the slope both
  equal to ß2, although the intercept are different.

20
Example Predictors of Writing Competence(interac
tion)

• For the model with interaction,
• y a ß1x1 ß2x2 ß3 x1 x2 e
• when 0 (male)
• average y score a ß2x2 ,
• when 1 (female)
• average y score a ß1 (ß2 ß3 ) x2
• With interaction, the lines not only have
  different intercepts but also have different
  slope (unless ß3 0).
• The change in average writing score when x2
  increases by 1 depends on gender x1.

21
14.2 Fitting a Model and Assessing Its Utility

• Suppose a multiple regression model includes a
  selected set of k predictor variables x1 , x2 ,
  xk
• y a ß1x1 ß2x2 ßk xk e
• It is then necessary to
• estimate the model coefficients,
• assess the models utility, and
• use the estimated model to make further
  inferences.
• A sample of n independent observations is
  selected, with each observation consists of k 1
  numbers (x1 , x2 , xk , y).

22
Definition The Least Squares Estimates

• According to the principle of least squares, the
  fit of a particular estimated regression function
  
• a b1x1 b2x2 bk xk
• to the observed data is measured by the sum of
  squared deviations between the observed y values
  and the y values predicted by the estimated
  function
• ? y - (a b1x1 b2x2 bk xk ) 2
• The least squares estimates of a, ß1, ß2, , ßk
  are those values of a, b1, b2, , bk that make
  this sum of squared deviations as small as
  possible.

23
Example Graduation Rates

• One way colleges measure success is by graduation
  rates. We are considering the following
  variables
• y six-year graduation rate
• x1 median SAT score of students accepted to
  the college
• x2 student-related expense per full-time
  student (in dollars)
• x3 1 if college has only female or only male
  students
• 0 if college has both male and female
  students
• The data in next slide represent a random
  sample of 22 colleges selected from the 1037
  colleges in US with enrollments less than 5000
  students.
• Fit a linear regression model to describe the
  relationship between y (the six year graduation
  rate) and the three predictor variables.

24
(No Transcript)
25
Example Graduation Rate at Small Colleges

• In SPSS enter the data.
• In Analyze, choose the Regression, and then
  Linear.
• Enter y as the dependent and x1, x2, x3 as
  independent
• Click Statistics button, and a window opens.
  Choose Estimates and Model Fit, which should
  be the default. Click Continue.
• We are back to the Linear Regression window.
  Click OK.
• We can see the Output - SPSS Viewer. Pull the
  screen down we can see the coefficients. These
  are the constant term and coefficients of x
  variables
• y -0.3906 0.0007602x1 0.000006969x2
  0.125 x3

26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Example Graduation Rate at Small Colleges

• The estimated mean value of y for specified x1,
  x2, and x3 values are y -0.3906 0.0007602x1
  0.000006969x2 0.125 x3
• Use the regression function to estimate the mean
  six-year graduation rate of coed college with a
  median SAT of 1000 and an expenditure per
  full-time student of 11,000.
• What is the six-year graduation rate for a
  particular college with a median SAT of 1000 and
  an expenditure per full-time student of 11,000?

34
Is the Model Useful? The F Test

• In the multiple regression model with regression
  function
• y a ß1x1 ß2x2 ßk xk, if all k
  coefficients ß1, ß2, , ßk are 0, there is no
  useful linear relationship between y and any of
  the predictor variables x1, x2, , xk in the
  model.
• We need to confirm the models utility through a
  formal test procedure.
• When all k ßi's are 0 in the model y a ß1x1
  ß2x2 ßkxk e and when the distribution of
  e is normal with mean 0 and variance s2 for any
  particular values of x1, x2, , xk, the model
  utility test for multiple regression has an F
  probability distribution based on df1 k (the
  number of model predictors), and df2 n ( k
  1 ) (n is the sample size).

35
F Distribution for df14, df2 6
36
Hypotheses Test

• The null hypothesis, denoted by H0, is a claim
  about population characteristics that is
  initially assumed to be true.
• The alternative hypothesis, denoted by Ha, is the
  competing claim.
• The two possible conclusions are reject H0 , or
  fail to reject H0.
• a is a pre-determined significance level, which
  is the probability of rejecting H0 when H0 is
  true.
• P-value is the area under the associated F curve
  to the right of the calculated F value.
• H0 should be rejected if P-value a.
• H0 should not be rejected if P-value gt a.

37
The F Test for Utility of the Model y a ß1x1
ß2x2 ßkxk e

• Null hypothesis H0 ß1 ß2 ßk 0 (No
  useful linear relationship between y and any of
  the predictors.)
• Alternative hypothesis Ha At least one of ß1,
  ß2, , ßk is not 0. (A useful linear relationship
  between y and at least one of the predictors.)
• Test statistics F (F is defined by a relatively
  complicated formula, but the calculated value of
  F is also available from the output of statistics
  software, such as SPSS.)
• Reject H0 if P-value a.
• Fail to reject H0 if P-value gt a.

38
F Test for Utility of the Regression Model for
the Graduation Rate of Small Colleges

• The model is y a ß1x1 ß2x2 ßkx3 e where
  y six-year graduation rate, x1 median SAT
  score, x2 expenditure per full time student,
  and x3 is an indicator if the college is coed (0)
  or not (1).
• H0 ß1 ß2 ß3 0, Ha At least one of ß1, ß2,
  ß3 is not 0.
• The pre-determined significance level a .05.
• Test statistics F 37.164 (from the
  Output-SPSS Viewer when we used SPSS to
  estimate the regression model.)
• df1 k 3, and df2 n ( k 1 ) 22 (31)
  18. Use the F-curve Table to find P lt .001
• Reject H0 because P lt a. We thus confirm the
  utility of the model.

39

• Exercise The data in the table includes the
  price, calorie content, protein content, and fat
  content for a sample of 19 energy bars.
• Fit a multiple regression model to describe the
  relationship between price and the three
  predictors calories, protein content, and fat
  content.
• Carry out the model utility test to determine if
  the model is useful for predicting price. Use a
  .01.

40
Chapter 14 Assignment

• Complete the exercise problem about the price of
  energy bars on the previous slide.
• Problems14.2, 14.5, 14.7, 14.9 on page 640 642.
• Problems 14.15, 14.23, 14.27, 14.29 on page 655
  659.
• Please turn in Chapter 14 assignment before
  Exam 3. No late assignment will be accepted.