Linear Algebra Review

1
Linear Algebra Review
CS479/679 Pattern RecognitionDr. George Bebis
2
n-dimensional Vector

• An n-dimensional vector v is denoted as follows
• The transpose vT is denoted as follows

3
Inner (or dot) product

• Given vT (x1, x2, . . . , xn) and wT (y1, y2,
  . . . , yn), their dot product defined as
  follows

(scalar)
or
4
Orthogonal / Orthonormal vectors

• A set of vectors x1, x2, . . . , xn is orthogonal
  if
• A set of vectors x1, x2, . . . , xn is
  orthonormal if

5
Linear combinations

• A vector v is a linear combination of the vectors
  v1, ..., vk if
• where c1, ..., ck are constants.
• Example vectors in R3 can be expressed as a
  linear combinations of unit vectors i (1, 0,
  0), j (0, 1, 0), and k (0, 0, 1)

6
Space spanning

• A set of vectors S(v1, v2, . . . , vk ) span
  some space W if every vector in W can be written
  as a linear combination of the vectors in S
• - The unit vectors i, j, and k span R3

w
7
Linear dependence

• A set of vectors v1, ..., vk are linearly
  dependent if at least one of them is a linear
  combination of the others.

(i.e., vj does not appear on the right side)
8
Linear independence

• A set of vectors v1, ..., vk is linearly
  independent if no vector can be represented as a
  linear combination of the remaining vectors,
  i.e.

Example
9
Vector basis

• A set of vectors (v1, ..., vk) forms a basis in
  some vector space W if
• (1) (v1, ..., vk) are linearly independent
• (2) (v1, ..., vk) span W
• Standard bases

R2 R3 Rn
10
Matrix Operations

• Matrix addition/subtraction
• Matrices must be of same size.
• Matrix multiplication

m x p
q x p
m x n
Condition n q
11
Identity Matrix
12
Matrix Transpose
13
Symmetric Matrices
Example
14
Determinants
2 x 2
3 x 3
n x n
Properties
15
Matrix Inverse

• The inverse A-1 of a matrix A has the property
• AA-1A-1AI
• A-1 exists only if
• Terminology
• Singular matrix A-1 does not exist
• Ill-conditioned matrix A is close to being
  singular

16
Matrix Inverse (contd)

• Properties of the inverse

17
Matrix trace
Properties
18
Rank of matrix

• Equal to the dimension of the largest square
  sub-matrix of A that has a non-zero determinant.
• Example

has rank 3
19
Rank of matrix (contd)

• Alternative definition the maximum number of
  linearly independent columns (or rows) of A.

Example
i.e., rank is not 4!
20
Rank of matrix (contd)
21
Eigenvalues and Eigenvectors

• The vector v is an eigenvector of matrix A and ?
  is an eigenvalue of A if
• i.e., the linear transformation implied by A
  cannot change the direction of the eigenvectors
  v, only their magnitude.

(assume non-zero v)
22
Computing ? and v

• To find the eigenvalues ? of a matrix A, find the
  roots of the characteristic polynomial

Example
23
Properties

• Eigenvalues and eigenvectors are only defined for
  square matrices (i.e., m n)
• Eigenvectors are not unique (e.g., if v is an
  eigenvector, so is kv)
• Suppose ?1, ?2, ..., ?n are the eigenvalues of A,
  then

24
Matrix diagonalization

• Given an n x n matrix A, find P such that
• P-1AP? where ? is diagonal
• Take P v1 v2 . . . vn, where v1,v2 ,. . . vn
  are the eigenvectors of A

25
Matrix diagonalization (contd)
Example
26
Are all n x n matrices diagonalizable P-1AP ?

• Only if P-1 exists (i.e., P must have n linearly
  independent eigenvectors, that is, rank(P)n)
• If A is diagonalizable, then the corresponding
  eigenvectors v1,v2 ,. . . vn form a basis in Rn

27
Matrix decomposition

• Let us assume that A is diagonalizable, then A
  can be decomposed as follows

28
Special case symmetric matrices

• The eigenvalues of a symmetric matrix are real
  and its eigenvectors are orthogonal.

P-1PT
APDPT