Advances in Random Matrix Theory stochastic eigenanalysis

1
Advances in Random Matrix Theory(stochastic
eigenanalysis)

• Alan Edelman
• MIT Dept of Mathematics,
• Computer Science AI Laboratories
• Monday June 6, 2005

2
Spreading the word
3
Scalars, Vectors, Matrices

• Mathematics Notation power less ink!
• Computation Use those caches!
• Statistics Classical, Multivariate, ?
• Modern Random
  Matrix Theory
• The Stochastic Eigenproblem
• Mathematics of probabilistic
  linear algebra
• Emerging Computational
  Algorithms
• Emerging Statistical Techniques

4
Outline

• In the beginning (Wigner)
• The classical cases (Hermite, Laguerre, Jacobi)
• Latest Greatest Theory (Free Probability)
• The Calculator
• Real World Networks
• Fancy Example
• Modern Special Functions

5
Wigners Semi-Circle

• The classical most famous rand eig theorem
• Let S random symmetric Gaussian
• MATLAB Arandn(n) S( AA)/2
• S known as the Hermite Ensemble
• Normalized eigenvalue histogram is a semi-circle
• Precise statements require n?? etc.

6
Wigners Semi-Circle

• The classical most famous rand eig theorem
• Let S random symmetric Gaussian
• MATLAB Arandn(n) S( AA)/2
• S known as the Hermite Ensemble
• Normalized eigenvalue histogram is a semi-circle
• Precise statements require n?? etc.

n x n iid standard normals
7
Wigners Semi-Circle

• The classical most famous rand eig theorem
• Let S random symmetric Gaussian
• MATLAB Arandn(n) S( AA)/2
• S known as the Hermite Ensemble
• Normalized eigenvalue histogram is a semi-circle
• Precise statements require n?? etc.

8
From Finite to Infinite
9
From Finite to Infinite
? Gaussian (m1)
10
From Finite to Infinite
? Gaussian (m1)
Wiggly
11
From Finite to Infinite
? Gaussian (m1)
Wiggly
Wigner?
12
The Classical CasesGrandn(n) or randn(m,n)

• Hermite AG? S(AA)/2
• Laguerre A G? S AA (Sample Covariance)
• Jacobi A G1?1 B G2?2 S(AA)-1(BB)
• Matrix analogs of Gaussian, Chi-square, Beta

13
The flipping coins example

• Classical Probability Coin 1 or -1 with p.5

50
50
50
50
y
x
-1 1
-1 1
xy
-2 0
2
14
The flipping coins example

• Classical Probability Coin 1 or -1 with p.5

Free
50
50
50
50
eig(B)
eig(A)
-1 1
-1 1
eig(AQBQ)
-2 0
2
15
Very powerful theory!

• Allows great generality over the classical cases

16
Haar or not Haar?
Uniform Distribution on orthogonal
matrices Gram-Schmidt or Q,RQR(randn(n))
17
Haar or not Haar?
Uniform Distribution on orthogonal
matrices Gram-Schmidt or Q,RQR(randn(n))
?
Eigenvalues Wrong
18
Longest Increasing Subsequence(n4)
Green 4 Yellow 3 Red 2 Purple 1
19
Random Matrix Result

• Permutations on 1..n with longest increasing
  subsequence k is
• E ( tr(Qk)2n) . The 2nth moment of the
  absolute trace of random kxk orthogonal matrices
• Longest increasing subsequence is the parallel
  complexity of an upper triangular solve with
  sparsity given by
• Uij(p) ?0 if p(i)p(j) and ij

20
Free Probability vs Classical Probability
21
Free Probability vs Classical Probability
anything
22
Random Matrix Calculator
23
Example Real World graphs
24
Example Real World graphs
25
Example Real World graphs
Data
26
Random Matrix Calculator
27
How to use calculator
28
Steps 1 and 2
29
Steps 3 and 4
30
Steps 5 and 6
31
Largest Eigenvalue of Hermite
32
Everyones Favorite Tridiagonal





33
Everyones Favorite Tridiagonal
1 (ßn)1/2







34
Stochastic Operator Limit



35
Spacings of eigs of AA
36
Riemann Zeta Zeros
37
Painlevé Equations
38
Matrix Statistics

• Many Worked out in 1950s and 1960s
• Muirhead Aspects of Multivariate Statistics
• Are two covariance matrices equal?
• Does my matrix equal this matrix?
• Is my matrix a multiple of the identity?
• Answers Require Computation of
• Hypergeometrics of Matrix Argument
• Long thought Computationally Intractible

39
Multivariate Orthogonal PolynomialsHypergeometr
ics of Matrix Argument

• The important special functions of the 21st
  century
• Begin with w(x) on I
• ? p?(x)p?(x) ?(x)ß ?i w(xi)dxi d??
• Jack Polynomials orthogonal for w1 on the unit
  circle. Analogs of xm

40
Multivariate Hypergeometric Functions
41
Multivariate Hypergeometric Functions
42
Plamens clever idea
43
Smallest eigenvalue statistics
Arandn(m,n) hist(min(svd(A).2))
44
Symbolic MOPS applications
Arandn(n) S(AA)/2 trace(S4)
det(S3)
45
Matrix Functions and Factorizations

• e.g. f(A)A2 or L,Ulu(A) or Q,Rqr(A)
• U, R n(n1)/2 parameters
• L, Q n(n-1)/2 parameters
• Q globally (Householder)
• Q locally (tangent space Qantisym )
• The Jacobian or df or linearization is n2
  x n2
• fS?S2 (sym) df is n(n1)/2 x n(n1)/2
• fQ?Q2 (orth) df is n(n-1)/2 x n(n-1)/2

46
Condition number of a matrix function or
factorization
Jacobian Det J ?si(df)det(df) Example 1
f(A)A2 df(A) kron(I,A)kron(AT ,I) Example 2
f(A)A-1 df(A)-kron(A-T,A-1) df(A)A-12
? ?A A-1
47
Matrix Factorization Jacobians
General
ALU AU?VT AX?X-1
AQR AQS (polar)
? uiin-i
? riim-i
? (?i2- ?j2)
? (?i?j)
? (?i-?j)2
Sym
Orthogonal
?sin(?i ?j)sin (?i- ?j)
Tridiagonal
TQ?QT
? (ti1,i)/ ?qi
48
Numerical Analysis Condition Numbers

• ?(A) condition number of A
• If AU?V is the svd, then ?(A) ?max/?min .
• Alternatively, ?(A) ?? max (AA)/?? min (AA)
• One number that measures digits lost in finite
  precision and general matrix badness
• Smallgood
• Largebad
• The condition of a random matrix???

49
Von Neumann co.

• Solve Axb via x (AA) -1A b
• M ?A-1
• Matrix Residual AM-I2
• AM-I2lt 200?2 n ?
• How should we estimate ??
• Assume, as a model, that the elements of A are
  independent standard normals!

?
50
Von Neumann co. estimates (1947-1951)

• For a random matrix of order n the expectation
  value has been shown to be about n
• Goldstine, von Neumann
• we choose two different values of ?, namely n
  and ?10n
• Bargmann, Montgomery, vN
• With a probability 1 ? lt 10n
• Goldstine, von Neumann

X ?
51
Random cond numbers, n??
Distribution of ?/n
Experiment with n200
52
Conclusions
Linear algebra Statistics Theory Free
Probability Theory Multivariate Special
Functions Tools The Free Calculator Tools
Algorithms for Multivariate Special
Functions Other Computational Tools not mentioned
today
53
Tidbit of interest to Matrix Computations
Audience

• Condition Numbers and Jacobians of Matrix
  Functions and Factorizations or
• What is matrix calculus??

54
Tidbit of interest to Matrix Computations
Audienceand pure mathematicians!

• The most analytical random matrices seen from on
  high

55
Same structure everywhere!
Orthog Matrix MATLAB (Arandn(n)
Brandn(n))
56
Same structure everywhere!
Orthog Matrix Weight Stats
Graph Theory SymSpace
57
Tidbit of interest to Matrix Computations
Audienceand combinatorists!

• The longest increasing subsequence

58
Tidbit!

• Random Tridiagonalization leads to eigenvalues of
  billion by billion matrix!

59
sym matrix to tridiagonal form
Same eigenvalue distribution as AA O(n)
storage !! O(n) compute
60
General beta
beta 1 reals 2 complexes 4 quaternions
Bidiagonal Version corresponds To Wishart
matrices of Statistics
61
MATLAB

• beta1 n1e9 opts.disp0opts.issym1
• alpha10 kround(alphan(1/3)) cutoff
  parameters
• dsqrt(chi2rnd( beta(n-1(n-k-1))))'
• Hspdiags( d,1,k,k)spdiags(randn(k,1),0,k,k)
• H(HH')/sqrt(4nbeta)
• eigs(H,1,1,opts)

62
Tricks to get O(n9) speedup

• Sparse matrix storage (Only O(n) storage is used)
  
• Tridiagonal Ensemble Formulas (Any beta is
  available due to the tridiagonal ensemble)
• The Lanczos Algorithm for Eigenvalue Computation
  ( This allows the computation of the extreme
  eigenvalue faster than typical general purpose
  eigensolvers.)
• The shift-and-invert accelerator to Lanczos and
  Arnoldi (Since we know the eigenvalues are near
  1, we can accelerate the convergence of the
  largest eigenvalue)
• The ARPACK software package as made available
  seamlessly in MATLAB (The Arnoldi package
  contains state of the art data structures and
  numerical choices.)
• The observation that if k 10n1/3 , then the
  largest eigenvalue is determined numerically by
  the top k k segment of n. (This is an
  interesting mathematical statement related to the
  decay of the Airy function.)

63
Tidbit of interest to Matrix Computations
AudienceStochastic Eigenequations

• Continuous vs Discrete
• Diff Eqns Matrix Comps Cont Eig Matrix
  Eigs
• Add probability
• Stochastic Differential Equations Stochastic
  Eigenequations
• Finite Random Matrix Theory

64
Stochastic Operator
65
Tidbit

• eig(AB) eig(A) eig(B) ?????

66
Summary

• Linear Algebra Randomness !!!

67
Mops (Dumitriu etc.) Symbolic
68
Symbolic MOPS applications
ß3 hist(eig(S))
69
(No Transcript)
70
Spacings

• Take a large collection of consecutive
  zeros/eigenvalues.
• Normalize so that average spacing 1.
• Spacing Function Histogram of consecutive
  differences (the (k1)st the kth)
• Pairwise Correlation Function Histogram of all
  possible differences (the kth the jth)
• Conjecture These functions are the same for
  random matrices and Riemann zeta

71
Largest Eigenvalue Plots
72
MATLAB

• beta1 n1e9 opts.disp0opts.issym1
• alpha10 kround(alphan(1/3)) cutoff
  parameters
• dsqrt(chi2rnd( beta(n-1(n-k-1))))'
• Hspdiags( d,1,k,k)spdiags(randn(k,1),0,k,k)
• H(HH')/sqrt(4nbeta)
• eigs(H,1,1,opts)

73
Open Problems
The distribution for general beta Seems to be
governed by a convection-diffusion equation
74
Random matrix tools!
75
Tidbit of interest to Matrix Computations
Audienceand combinatorists!

• The longest increasing subsequence

76
Finite n

• n10
  n25
• n50
  n100

77
Condition Number Distributions
Real n x n, n?8

• Generalizations
• ß 1real, 2complex
• finite matrices
• rectangular m x n

Complex n x n, n?8
78
Condition Number Distributions
Square, n?8 P(?/nß gt x) (2ßß-1/G(ß))/xß
(All Betas!!) General Formula P(?gtx) Cµ/x
ß(n-m1), where µ ß(n-m1)/2th moment of the
largest eigenvalue of Wm-1,n1 (ß) and C
is a known geometrical constant. Density for the
largest eig of W is known in terms of
1F1((ß/2)(n1), ((ß/2)(nm-1) -(x/2)Im-1) from
which µ is available Tracy-Widom law applies
probably all beta for large m,n. Johnstone shows
at least beta1,2.
Real n x n, n?8
Complex n x n, n?8