Multiple Regression II

1
Multiple Regression II
2
Multicollinearity

• In multiple regression, the estimates of each
  regression coefficient is dependent on the values
  of all of the independent variables in the model,
  not just the variable associated with the
  parameter.
• Thus, if Y B0 B1 X1 B2 X2 e
• Then the estimate of B1 will be dependent on the
  value of both X1 and X2 similarly, the estimate
  of B2 will be dependent on the estimate of X2 and
  X1.
• When B1 is conditional on X2 in addition to X1,
  we say that the regression is multicollinear.

3
Motivation

• Multicollinearity provides the motivation for why
  we estimate a multiple regression model rather
  than simply looking at a series of simple
  regression models, each with the same dependent
  variable.
• 2) The fact that the value of B1 is conditional
  on X1 and X2 underlies the utility of regression
  as an alternative to scientific controls.

4
Some MathDo not copy this slideit is just for
illustration.

• To understand multicollinearity, it is useful to
  start with the OLS estimates for B1 and B2 in the
  model
• Y B0 B1 X1 B2 X2 e
• Let x1 X1 Mean(X1), x2 X2 Mean(X2), y Y
  Mean(Y)
• B1 (?x1iyi) (?x2i2) - (?x2iyi) (?x1ix2i)
• (?x1i2)(?x2i2) (?x1ix21)2
• (A symmetric result exists for B2)
• The key thing to notice about the estimate of B1
  is that the covariance between X2 and Y and the
  covariance between X1 and X2 both influence B1.

5
Example

• Example from the Crime and Temperature Data Set

6
The intuition behind the OLS estimators.

• Suppose you have the regression model
• Y B0 B1X1 B2 X2 ei
• ICBST the OLS estimate of B1 can be estimated by
  the following procedure
• 1) Get OLS estimates for Y ?0 ?1X2 ui
  (save the residuals ui)
• 2) Get OLS estimates for X1 ?0 ?1 X2 vi
  (save the residuals vi)
• 3) Get OLS estimates for ui ??0 ?1 vi wi
• Then, the OLS estimate of B1 is equal to the OLS
  estimate of ?1.
• The intuition behind the procedure is that we can
  estimate the effect of X1 on Y in step 3, purged
  of the effects of X2 on Y (step 1), and of the
  effects of X2 on X2 (step 2).

7
Concerns about Multicollinearity

• As a general rule, the presence of correlated
  independent variables is not a concern. However,
  correlated independent variables can be a serious
  problem in at least two situations.
• 1) Your independent variables are perfectly
  correlated. In this case, OLS will not provide a
  unique estimate for your regression coefficients.
  To see this, suppose your model was
• Y B0 B1 X1 B2 X2, but X1 X2
• In this case, B1 and B2 could take on any set of
  values, so long as B1 B2 equaled the true
  value of B1
• 2) You include two measures of the same concept.
  In this case, OLS estimates of both regression
  coefficients may be attenuated because you have
  two independent variables measuring one cause and
  effect relationship. In effect B1X1 B2 X2
  together would, in effect, measure the effect
  that could be attributed just to X1.

8
Topic 2. Dummy Variables

• Sometimes, you have categorical variables that
  you would like to use as independent variables in
  a regression.
• Examples might include gender, religion, race,
  etc..
• For dichotomous variables, you simply include a
  variable that is coded as 1 for one category and
  zero for the other.
• e.g. female 1 and male 0.
• In this case, the regression coefficient is
  interpreted as the difference in the dependent
  variable between men and women.

9

• Note to self
• Draw a scatterplot on the board using Xs to
  denote men and Os to denote women, where men the
  distribution of men is shifted above the
  distribution of women, but the slope of the line
  is essentially the same.

10
Dummy Variables cont.

• Polychotomous variables are trickier than
  dichotomous variables.
• One variable would not be very effective at
  measuring the effect of a polychotomous variable
  whose categories lack a clear ordering.
• For example, consider the regression model
• Income B0 B1 Race B2 Yrs Education B3
  Yrs Experience
• What is wrong with this model?

11
Dummy Variables cont.

• If there were only two races, then we could
  estimate the model with race coded such that
  blacks 1 and whites 0.
• Income B0 B1 Race B2 Yrs Education B3 Yrs
  Experience
• However, there are multiple categories of race,
  and we wouldnt want to say that, Asians had a
  bit more race than whites but less race than
  blacks.
• Instead, we use a set of dummy variables in
  order to describe the polychotomous variable.
  Each dummy variable in the set describes whether
  or not an individual belongs to a certain
  category in the model.

12
Dummy Variables cont.

• Example to describe a polychotomous variable for
  race that contained the categories black, white ,
  Asian, and Latino, we could use the following
  variables
• BlackVar 1 for blacks, 0 for all others
• WhiteVar 1 for whites, 0 for all others
• LatinoVar 1 for Latinos, 0 for all others
• Each of these variables would then be included in
  a regression model
• Income B0 B1 BlackVar B2 WhiteVar B3
  LatinoVar
• B4 Yrs Education B5 Yrs Experience
• Why dont we include a variable for Asians?
• The regression coefficients are then interpreted
  as the effect of being in Category X as compared
  to being in the excluded group. Therefore, in the
  above example, B1 gives the increase/decrease in
  income for blacks relative to Asians.