Review of Linear Algebra 10-725 - Optimization 1/14/1

1
Review of Linear Algebra

• 10-725 - Optimization1/14/10 Recitation
• Sivaraman Balakrishnan

2
Outline

• Matrix subspaces
• Linear independence and bases
• Gaussian elimination
• Eigen values and Eigen vectors
• Definiteness
• Matlab essentials
• Geoffs LP sketcher
• linprog
• Debugging and using documentation

3
Basic concepts

• Vector in Rn is an ordered set of n real numbers.
• e.g. v (1,6,3,4) is in R4
• A column vector
• A row vector
• m-by-n matrix is an object with m rows and n
  columns, each entry filled with a real
  (typically) number

4
Basic concepts - II

• Vector dot product
• Matrix product

5
Matrix subspaces

• What is a matrix?
• Geometric notion a matrix is an object that
  transforms a vector from its row space to its
  column space
• Vector space set of vectors closed under scalar
  multiplication and addition
• Subspace subset of a vector space also closed
  under these operations
• Always contains the zero vector (trivial
  subspace)

6
Row space of a matrix

• Vector space spanned by rows of matrix
• Span set of all linear combinations of a set of
  vectors
• This isnt always Rn example !!
• Dimension of the row space number of linearly
  independent rows (rank)
• Well discuss how to calculate the rank in a
  couple of slides

7
Null space, column space

• Null space it is the orthogonal compliment of
  the row space
• Every vector in this space is a solution to the
  equation
• Ax 0
• Rank nullity theorem
• Column space
• Compliment of rank-nullity

8
Linear independence

• A set of vectors is linearly independent if none
  of them can be written as a linear combination of
  the others
• Given a vector space, we can find a set of
  linearly independent vectors that spans this
  space
• The cardinality of this set is the dimension of
  the vector space

9
Gaussian elimination

• Finding rank and row echelon form
• Applications
• Solving a linear system of equations (we saw this
  in class)
• Finding inverse of a matrix

10
Basis of a vector space

• What is a basis?
• A basis is a maximal set of linearly independent
  vectors and a minimal set of spanning vectors of
  a vector space
• Orthonormal basis
• Two vectors are orthonormal if their dot product
  is 0, and each vector has length 1
• An orthonormal basis consists of orthonormal
  vectors.
• What is special about orthonormal bases?
• Projection is easy
• Very useful length property
• Universal (Gram Schmidt) given any basis can find
  an orthonormal basis that has the same span

11
Matrices as constraints

• Geoff introduced writing an LP with a constraint
  matrix
• We know how to write any LP in standard form
• Why not just solve it to find opt?

12
A special basis for square matrices

• The eigenvectors of a matrix are unit vectors
  that satisfy
• Ax ?x
• Example calculation on next slide
• Eigenvectors are orthonormal and eigenvalues are
  real for symmetric matrices
• This is the most useful orthonormal basis with
  many interesting properties
• Optimal matrix approximation (PCA/SVD)
• Other famous ones are the Fourier basis and
  wavelet basis

13
Eigenvalues

• (A ?I)x 0
• ? is an eigenvalue iff det(A ?I) 0
• Example

14
Projections (vector)
(2,2,2)
b (2,2)
(0,0,1)
(0,1,0)
(1,0,0)
a (1,0)
15
Matrix projection

• Generalize formula from the previous slide
• Projected vector (QTQ)-1 QTv
• Special case of orthonormal matrix
• Projected vector QTv
• Youve probably seen something very similar in
  least squares regression

16
Definiteness

• Characterization based on eigen values
• Positive definite matrices are a special
  sub-class of invertible matrices
• One way to test for positive definiteness is by
  showing
• xTAx gt 0 for all x
• A very useful example that youll see a lot in
  this class
• Covariance matrix

17
Matlab Tutorial - 1

• Linsolve
• Stability and condition number
• Geoffs sketching code might be very useful for
  HW1 ?

18
Matlab Tutorial - 2

• Linprog Also, very useful for HW1 ?
• Also, covered debugging basics and using Matlab
  help

19
Extra stuff

• Vector and matrix norms
• Matrix norms - operator norm, Frobenius norm
• Vector norms - Lp norms
• Determinants
• SVD/PCA