NOTES ON MULTIPLE

1
NOTES ON MULTIPLE REGRESSION USING MATRICES
Tony E. Smith
ESE 502 Spatial Data Analysis
? Multiple Regression
? Matrix Formulation of Regression
? Applications to Regression Analysis
2
SIMPLE LINEAR MODEL
? Data
? Parameters
? Model
3
SIMPLE REGRESSION ESTIMATION
? Estimate Conditional Mean
? Data Points
? Predicted Value
where
4
STANDARD LINEAR MODEL
? Data
? Parameters
? Model
5
STANDARD LINEAR MODEL (k 2)
? Data
? Parameters
? Model
6
REGRESSION ESTIMATION (for k 2)
? Data Points
? Predicted Value
where
7
MATRIX REPRESENTATION OF THE STANDARD LINEAR MODEL
? Vectors and Matrices
? Matrix Reformulation of the Model
8
LINEAR TRANSFORMATIONS IN ONE DIMENSION
? Linear Function
? Graphic Depiction
9
LINEAR TRANSFORMATIONS IN TWO DIMENSIONS
? Linear Transformation
10
? Graphical Depiction of Linear Transformation
?
?
?
?
?
?
?
?
11
SOME MATRIX CONVENTIONS
? Transposes of Vectors and Matrices
? Symmetric (Square) Matrices
? Important Example
12
? Row Representation of Matrices
? Column Representation of Matrices
13
? Inner Product of Vectors
? Matrix Multiplication
? Transposes
14
MATRIX REPRESENTATIONS OF LINEAR TRANSFORMATIONS
? For any Two-Dimensional Linear Transformation

with
15
? Graphical Depiction of Matrix Representation
?
?
?
?
?
?
?
?
16
? Inversion of Square Matrices (as Linear
Transformations)
17
DETERMINANTS OF SQUARE MATRICES
18
NONSINGULAR SQUARE MATRICES
?
?
?
?
?
?
19
LEAST-SQUARES ESTIMATION
? General Regression Matrices
? General Sum-of-Squares
20
DIFFERENTIATION OF FUNCTIONS
? General Derivative
?
? Example
?
?
21
PARTIAL DERIVATIVES
?
?
22
VECTOR DERIVATIVES
? Derivative Notation for
? Gradient Vector
23
TWO IMPORTANT EXAMPLES
? Linear Functions
? Quadratic Functions
24
? Quadratic Derivatives
? Symmetric Case
25
MINIMIZATION OF FUNCTIONS
? First-Order Condition
? Example
?
?
26
TWO-DIMENSIONAL MINIMIZATION
?
?
?
27
LEAST SQUARES ESTIMATION
? Solution for
28
NON-MATRIX VERSION (k 2)
? Data
? Beta Estimates
29
EXPECTED VALUES OF RANDOM MATRICES
? Random Vectors and Matrices
? Expected Values
30
EXPECTATIONS OF LINEAR FUNCTIONS OF RANDOM VECTORS
? Linear Combinations
? Linear Transformations
31
EXPECTATIONS OF LINEAR FUNCTIONS OF RANDOM
MATRICES
? Left Multiplication
? Right Multiplication (by symmetry of inner
products)
32
COVARIANCE OF RANDOM VECTORS
? Random Variables
? Random Vectors
33
COVARIANCE OF LINEAR FUNCTIONS OF RANDOM VECTORS
? Linear Transformations
( Left Mult )
( Right Mult )
? Linear Combinations
34
TRANSLATIONS OF RANDOM VECTORS
? Translation
? Means
? Covariances
35
RESIDUAL VECTOR IN THE STANDARD LINEAR MODEL
? Linear Model Assumption
? Residual Means
? Residual Covariances
36
MOMENTS OF BETA ESTIMATES
? Linear Model
? Mean of Beta Estimates
(Unbiased Estimator)
? Covariance of Beta Estimates
37
ESTIMATION OF RESIDUAL VARIANCE
? Residual Variance
? Residual Estimates
? Natural Estimate of Variance
? Bias-Correct Estimate of Variance
(Compensates for Least Squares)
38
ESTIMATION OF BETA COVARIANCE
? Beta Covariance Matrix
? Beta Covariance Estimates