Linear Least Squares

1
Linear Least Squares

• Based on Greenes Note 3

2
Vocabulary

• Some terms to be used in the discussion.
• Population characteristics and entities vs.
  sample quantities and analogs
• Residuals and disturbances
• Population regression line and sample regression
• Objective Learn about the conditional mean
  function. Estimate ? and ?2
• First step Mechanics of fitting a line
  (hyperplane) to a set of data

3
Fitting Criteria

• The set of points in the sample
• Fitting criteria - what are they
• LAD
• Least squares
• and so on
• Why least squares? (We do not call it ordinary
  at this point.)
• A fundamental result
• Sample moments are good estimators of
• their population counterparts
• We will spend the next few weeks using
  this principle and applying it to least squares
  computation.

4
An Analogy Principle

• In the population Ey X X? so
• Ey - X? X 0
• Continuing Exi ?i 0
• Summing, Si Exi ?i Si 0 0
• Exchange Si and E ESi xi ?i E X??
  0
• E X? (y - X?)
  0
• Choose b, the estimator of ? to mimic this
  population result i.e., mimic the population
  mean with the sample mean
• Find b such that
• As we will see, the solution is the least
  squares coefficient vector.

5
Population and Sample Moments

• We showed that E?ixi 0 and Covxi,?i
  0. If it is, and if EyX X?, then
• ? (Varxi)-1 Covxi,yi.
• This will provide a population analog to the
  statistics we compute with the data.

6
Example

• U. S. Gasoline Market, 1953-2004
• G Total U.S. gasoline expenditure
• PG Price index for gasoline
• Y Per capita disposable income
• Pnc Price index for new cars
• Puc Price index for used cars
• Ppt Price index for public transportation
• Pd Aggregate price index for consumer durables
• Pn Aggregate price index for consumer
  nondurables
• Ps Aggregate price index for consumer services
• Pop U.S. total population in thousands

7
Least Squares

• Example will be, yi Gi on
• xi a constant, PGi and Yi 1,Pgi,Yi
• Fitting criterion Fitted equation will be
• yi b1xi1 b2xi2 ... bKxiK.
• Criterion is based on residuals
• ei yi - b1xi1 b2xi2 ... bKxiK
• Make ei as small as possible.
• Form a criterion and minimize it.

8
Fitting Criteria

• Sum of residuals
• Sum of squares
• Sum of absolute values of residuals
• Absolute value of sum of residuals
• We focus on now and later

9
Least Squares Algebra
10
Least Squares Normal Equations
11
Least Squares Solution
12
Second Order Conditions
13
Does b Minimize ee?
14
Sample Moments - Algebra
15
Positive Definite Matrix
16
Algebraic Results - 1
17
Residuals vs. Disturbances
18
Algebraic Results - 2

• The residual maker M (I - X(XX)-1X)
• e y - Xb y - X(XX)-1Xy My
• MX 0 (This result is fundamental!)
• How do we interpret this result in terms of
  residuals?
• (Therefore) My MXb Me Me e
• (You should be able to prove this.
• y Py My, P X(XX)-1X (I - M).
• PM MP 0. (Projection matrix)
• Py is the projection of y into the column space
  of X. (New term?)

19
The M Matrix

• M I- X(XX)-1X is an nxn matrix
• M is symmetric M M
• M is idempotent MM M
• (just multiply it out)
• M is singular M-1 does not exist.
• (We will prove this later as a side result
  in another derivation.)

20
Results when X Contains a Constant Term

• X 1,x2,,xK
• The first column of X is a column of ones
• Since Xe 0, x1e 0 the residuals sum to
  zero.

21
Example (Cont.)

• y Xb e
• y G X 1 PG Y
• Using Stata
• Using GAUSS