Review of Matrix Algebra and Vector Spaces

1
Review of Matrix Algebra and Vector Spaces

• Lecture XXVI

2
Review of Elementary Matrix Algebra

• The material for this lecture is found in James
  R. Schott Matrix Analysis for Statistics (New
  York John Wiley Sons, Inc. 1997).
• A matrix A of size m x n is an m x n rectangular
  array of scalars

3

• It is sometimes useful to partition matrices into
  vectors.

4

• The sum of two identically dimensioned matrices
  can be expressed as

5

• In order to multiply a matrix by a scalar,
  multiply each element of the matrix by the
  scalar.
• In order to discuss matrix multiplication, we
  first discuss vector multiplication. Two vectors
  x and y can be multiplied together to form z (zx
  y) only if they are conformable. If x is of
  order 1 x n and y is of order n x 1, then the
  vectors are conformable and the multiplication
  becomes

6

• Extending this discussion to matrices, two
  matrices A and B can be multiplied if they are
  conformable. If A is order k x n and B is of
  order n x l. then the matrices are conformable.
  Using the partitioned matrix above, we have

7
(No Transcript)
8

• Theorem 1.1 Let a and b be scalars and A, B, and
  C be matrices. Then when the operations involved
  are defined, the following properties hold
• ABBA.
• (AB)CA(BC).
• a(AB)aAaB.
• (ab)AaAbA.
• A-AA(-A)(0).

9

• A(BC)ABAC.
• (AB)CACBC.
• (AB)CA(BC).
• The transpose of an m x n matrix is a n x m
  matrix with the rows and columns interchanged.
  The transpose of A is denoted A.

10

• Theorem 1.2 Let a and b be scalars and A and B be
  matrices. Then when defined, the following hold
• (aA)aA.
• (A)A.
• (aAbB)aAbB.
• (AB)BA.

11

• The trace is a function defined as the sum of the
  diagonal elements of a square matrix.

12

• Theorem 1.3 Let a be scalar and A and B be
  matrices. Then when the appropriate operations
  are defined, we have
• tr(A)tr(A).
• tr(aA)atr(A).
• tr(AB)tr(A)tr(B).
• tr(AB)tr(BA).
• tr(AA)0 if and only if A(0).

13

• Traces can be very useful in statistical
  applications. For example, natural logarithm of
  the normal distribution function can be written
  as
• Jan R. Magnus and Heinz Neudecker Matrix
  Differential Calculus with Applications in
  Statistics and Econometrics (New York John Wiley
  Sons, 1988) p. 314.

14

• The Determinant is another function of square
  matrices. In its most technical form, the
  determinant is defined as
• where the summation is taken over all
  permutations, (i1,i2,im) of the set of integers
  (1,m), and the function f(i1,i2,im) equals the
  number of transpositions necessary to change
  (i1,i2,im).

15

• In the simple case of a 2 x 2, we have two
  possibilities (1,2) and (2,1). The second
  requires one transposition. Under the basic
  definition of the determinant

16

• In the slightly more complicated case of a 3 x 3,
  we have six possibilities (1,2,3), (2,1,3),
  (2,3,1), (3,2,1), (3,1,2), (1,3,2). Each one of
  these differs from the previous one by one
  transposition. Thus, the number of
  transpositions are 0, 1, 2, 3, 4, 5. The
  determinant is then defined as

17
(No Transcript)
18

• A more straightforward definition involves the
  expansion down a column or across the row.
• In order to do this, I want to introduce the
  concept of principal minors.
• The principal minor of an element in a matrix is
  the matrix with the row and column of the element
  removed.
• The determinant of the principal minor times
  negative one raised to the row number plus the
  column number is called the cofactor of the
  element.

19

• The determinant is then the sum of the cofactors
  times the elements down a particular column or
  across the row

20

• In the three by three case

21

• Theorem 1.4 If a is a scalar and A is an m x m
  matrix, then the following properties hold
• AA.
• aAamA.
• If A is a diagonal matrix, then Aa11a22amm.
• If all elements of a row (or column) of A are
  zero, A0.
• If two rows (or columns) of A are proportional to
  one another, A0.

22

• The interchange of two rows (or columns) of A
  changes the sign of A.
• If all the elements of a row (or column) of A are
  multiplied by a, then the determinant is
  multiplied by a.
• The determinant of A is unchanged when a multiple
  of one row (or column) is added to another row
  (or column).

23
The Inverse

• Any m x m matrix A such that A?0 is said to be
  a nonsingular matrix and possesses an inverse
  denoted A-1.

24

• Theorem 1.6 If a is a nonzero scalar, and A and B
  are nonsingular m x m matrices, then
• (aA)-1a-1A-1.
• (A)-1(A-1).
• (A-1)-1A.
• A-1A-1.
• If Adiag(a11,amm), then A-1diag(a11-1,amm-1).
• If AA, then A-1(A-1).
• (AB)-1B-1A-1.

25

• The most general definition of an inverse
  involves the adjoint matrix (denoted A). The
  adjoint matrix of A is the transpose of the
  matrix of cofactors of A. By construction of the
  adjoint, we know that

26

• In order to see this identity, note that
• Focusing on the first point

27
(No Transcript)
28

• Given this expression, we see that

29
(No Transcript)
30
(No Transcript)
31
Rank of a Matrix

• The rank of a matrix is the number of linearly
  independent rows or columns. One way to
  determine the rank of any general matrix m x n is
  to delete rows or columns until the resulting r x
  r matrix has a nonzero determinant. What is the
  rank of the above matrix? If the above matrix had
  been

32

• note A0. Thus, to determine the rank, we
  delete the last row and column leaving

33

• The rank of a matrix A remains unchanged by any
  of the following operations, called elementary
  transformations
• The interchange of two rows (or columns) of A.
• The multiplication of a row (or column) of A by a
  nonzero scalar.
• The addition of a scalar multiple of a row (or
  column) of A to another row (or column) of A.

34
Orthogonal Matrices

• An m x 1 vector p is said to be a normalized
  vector or a unit vector if pp1. The m x 1
  vectors p1, p2,pn where n is less than or equal
  to m are said to be orthogonal if pipj0 for all
  i not equal to j. If a group of n orthogonal
  vectors are also normalized, the vectors are said
  to be orthonormal. An m x m matrix consisting
  of orthonormal vectors is said to be orthogonal.
  It then follows

35

• It is possible to show that the determinant of an
  orthogonal matrix is either 1 or 1.

36
Quadratic Forms

• In general, the a quadratic form of a matrix can
  be written as
• We are most often interested in the quadratic
  form xAx.

37

• Every symmetric matrix A can be classified into
  one of five categories
• If xAx gt 0 for all x ? 0, the A is positive
  definite.
• If xAx 0 for all x ? 0 and xAx0 for some x ?
  0, the A is positive semidefinite.
• If xAx lt 0 for all x ? 0 then A is negative
  definite.
• If xAx 0 for all x ? 0 and xAx0 for some x ?
  0, the A is negative semidefinite.
• If xAxgt0 for some x and xAxlt0 for some x, then
  A is indefinite.

38
Vector Spaces

• Definition 2.1. Let S be a collection of m x 1
  vectors satisfying the following
• If x1 e S and x2 e S, then x1x2 e S.
• If x e S and a is a real scalar, the ax e S.
• Then S is called a vector space in m-dimensional
  space. If S is a subset of T, which is another
  vector space in m-dimensional space, the S is
  called a vector subspace of T.

39

• Definition 2.2 Let x1,xn be a set of m x 1
  vectors in the vector space S. If each vector in
  S can be expressed as a linear combination of the
  vectors x1,xn, then the set x1,xn is said to
  span or generate the vector space S, and x1,xn
  is called a spanning set of S.

40
Linear Independence and Dependence

• Definition 2.6 The set of m x 1 vectors x1,xn
  is said to be a linearly independent if the only
  solution to the equation
• is the zero vector a1an0.

41
(No Transcript)
42

• This reduction implies that
• Or that the third column of the matrix is a
  linear combination of the first two.