HighDimensional Covariance Inference

1
High-DimensionalCovariance Inference

• Charles Curry
• University of California, Santa Cruz
• Joint Statistical Meetings, Seattle 2006

2
Covariance Inference Problems

• Many parameters
• Quadratic in the number of data dimensions
• Often have too little data to estimate
• Expensive computations
• Numerical instability of estimation
• Imprecise likelihoods, eigenvectors
• Difficult to specify prior information
• Conjugate Wishart prior has limited
  interpretability
• Reference priors are difficult to derive

3
Covariance Structures

• Basis functions
• Non-zero eigenvalues
• Corresponding eigenvectors
• Principal components analysis, EOFs
• Separability of dimensions
• Kronecker products of smaller matrices
• Natural subsets in data dimensions
• Stationary process models
• Variograms
• Correlation function families
• Process Convolution
• Higdon (1999), Non-stationary spatial modeling.

4
Data Sources for Covariance

• Observations, too few to estimate covariance
• Model forced runs, MIT 2D Model
• Assume covariance structure stable over model
  parameter space
• Model control runs
• Combine all of the above
• Higher and lower dimensional data
• NCEP Reanalysis, www.cdc.noaa.gov

5
Eigen-analysis

• Maximum likelihood estimators
• Algorithms for estimating eigenvalues and
  eigenvectors of the covariance matrix
• Estimates are biased
• Consistent, bias is large for small data sizes
• Largest eigenvalues are too large
• Smallest eigenvalues are too small
• No uncertainties associated with estimates
• Eigenvalues significantly non-zero?
• Only the first eigenvector is easily interpretable

6
Eigenvalue Selection

• How many eigenvalues to keep?
• Model selection problem
• Different model selection criteria give different
  answers
• When there is too little data, many model
  selection criteria select only the first
  eigenvalue
• Separate signal from noise

7
Bayesian Inference

• Simple multivariate normal model
• Number of measurements n
• Data dimensionality p

8
Separability

• Assumptions
• Dramatic Parameter Savings
• Matrix-Variate Normal Likelihood
• Simple independence Jeffreys prior
• Product of priors for a single covariance matrix

9
Correlation Functions

• Small number of parameters
• Stationary/Isotropic Process Assumptions
• Choose a family
• Power exponential simple to evaluate, degenerate
  continuity properties
• Matern good theoretical properties, difficult to
  evaluate and to build priors for parameters
• Uncertainty in family parameters
• Priors from Berger, De Olivera and Sansó (2001),
  Objective Bayesian analysis of spatially
  correlated data.

10
Sample Covariogram

• NCEP
• Latitude
• Longitude
• Elevation
• Stationarity

11
Combining Models

• Underlying unknown covariance
• Shared by all data sources
• Allowed to differ by a scale factor
• One observation
• Small number samples from control run
• Auto-correlation in the samples
• Many single-sample model runs at different
  parameter settings
• Build a statistically-equivalent model using
  non-parametric Gaussian process regression
• Kennedy and OHagan (2001), Bayesian calibration
  of computer models. Priors from Paulo (2005),
  Default priors for Gaussian processes.

12
Full Model

• Control run likelihood x1, , xn
• Gaussian process interpolator over forced model
  runs y1, , ym
• Observation x0, all p dimensional
• Gibbs sampling where possible, MH for process
• Lots of ignorables, interested in inference for
• Covariance model between p dimensions
• Parameter inference for model parameters

13
Data

• Climate diagnostics surface temperature
• Anomalies from 100 year mean
• 4 latitude zonal averages by 5 decadal averages
  (p 20)
• Unforced control run has 95 overlapping outputs
  (n95)
• Forced model run at 499 parameter settings
  non-uniformly sampled in 3D parameter space
• MIT 2D Model. Forest, et al (2002), Quantifying
  uncertainties in climate system properties.

14
Covariance Inference Results

• Model Runs
• Uncertainty in eigenvalues
• Overlapping posteriors indeterminacy

15
Covariance Inference Results

• Little uncertainty in eigenvectors corresponding
  to largest eigenvalues

16
Covariance Inference Results

• Too much uncertainty in other eigenvectors
• Dont trust eigenvectors associated with small
  eigenvalues

17
Eigenvector Interpretation

• Easy to interpret the first eigenvector
• Models vary in the amount of warming they show
  across the five decades
• Remainder must consider orthogonality and effect
  of uncertainty on posterior mean

18
MCMC Mixing

• Gibbs-able regression parameters, tight
  posteriors
• Almost full-conditional for covariance
• Good mixing on covariance with proper prior
• Tough to get decent mixing for interpolation
  process parameters

19
Current Focus

• Robust software for Bayesian covariance inference
  and multivariate response Gaussian process
  regression
• Extensions to larger data sets, dimensionality
• Upper-air diagnostic, p288
• Separability among dimensions latitude-time,
  latitude-pressure
• Parametrization and prior construction for
  covariance matrices

20
Future Work

• Build priors from high-resolution data
• NCEP multivariate response, standardized grid,
  combination of model and data
• Process convolutions for non-stationarity and
  computational tractability
• Focus on incremental updating, advantage of
  Bayesian approach
• Dependent component analysis