Anomaly Compensation and Cloud Clearing of AIRS Hyperspectral Data

1
Anomaly Compensation and Cloud Clearing of AIRS
Hyperspectral Data

• Presented at IEEE GRSS
• Boston Section Meeting
• August 24, 2005
• Choongyeun (Chuck) Cho

2
Overview I

• Problem statement
• Definition of anomaly
• Background
• Atmospheric InfraRed Sounder (AIRS), Advanced
  Microwave Souding Unit (AMSU), and Humidity
  Sounder for Brazil (HSB) instruments
• Signal characterization and reduction of
  artifacts
• Principal component analysis (PCA) and its
  variants
• Artifacts in AIRS data

3
Overview II

• Stochastic cloud-clearing
• Background and prior work
• Description of stochastic-clearing (SC) algorithm
• Validation of stochastic-clearing algorithm
• ECMWF
• Physical clearing
• Conclusion
• Summary / contributions
• Future work

4
Where Are We?

• Problem statement
• Definition of anomaly
• Background
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Validation of SC algorithm
• Conclusion

5
Problem Statement

• What is hyperspectral data?
• Hundreds or thousands of contiguous channels
• What is anomaly?
• Defined as an unwanted spatial or spectral
  signature, statistically distinct from its
  surrounding
• Given X and a priori information about ?, what is
  best estimate of ? or X?
• A priori info about anomaly can have different
  forms
• Spectral statistical description, or usually
  ensembles
• Spatial structure or texture
• Joint spatial/spectral description

6
Examples of Anomalies

• Anomalies of AIRS data discussed in this talk
• Instrumental noise
• Noisy channels
• AIRS data exhibits consistently noisy channels
• Scan-line miscalibration
• Resulting in striping patterns
• Cloud contamination
• Clouds generally make IR observations colder
• Compensation for cloud impact is critical for
  accurate retrieval

7
Where Are We?

• Problem statement
• Background
• AIRS/AMSU/HSB instruments
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Validation of SC algorithm
• Conclusion

8
AIRS/AMSU/HSB Instruments on Aqua
AIRS/HSB

• Atmospheric InfraRed Sounder (AIRS)
• 2378-channel infrared spectrometer covering
  3.7-15.4µm
• 1.1 FOV (13.5km at nadir)
• Advanced Microwave Sounding Unit (AMSU)
• 15 microwave channels
• 3.3 FOV (40.5km at nadir)
• Humidity Sounder for Brazil (HSB)
• 1.1 FOV (13.5km at nadir, same as AIRS)
• 4 microwave channels (150,190 GHz)
• Scan motor failure since Feb 2003

AMSU golfball
9
Sample AIRS Brightness Temperatures
AIRS sample spectra for 3-by-3 FOVs
AIRS sample image
Data August 21, 2003 near Great Lakes
10
Where Are We?

• Problem statement
• Background
• Signal characterization and reduction of
  artifacts
• Principal component analysis (PCA) and its
  variants
• Artifacts in AIRS data
• Stochastic cloud-clearing
• Validation of SC algorithm
• Conclusion

11
Principal Component Analysis

• Useful to characterize multivariate signal, and
  reduce dimensionality
• Can be defined recursively (x m-dim multivariate
  signal)
• Solution y WTx where W w1w2wn, wi are
  eigenvectors of CXX which is often estimated
• To reduce dimension, n ltlt m
• The resulting reconstructed signal
  (PC filter)

12
Noise-Adjusted Principal Component

• Principal components are sensitive to arbitrary
  scaling
• Noise-adjusted PC (NAPC)
• Normalize data before applying PCA
• Guarantees maximum SNR
• What if noise variances are not known?
• Need to be estimated using blind signal
  separation (BSS) technique such as Iterative
  Order and Noise (ION) estimation

13
Noise-Adjusted Principal Component

• Typical variances versus PC index plot
  (truncated at 400)
• NAPC shows sharper break than PC
• 6 NAPCs explain 99.8 of variances

Cumulative explained variance
Scree plot for AIRS data
Variance
14
Artifacts in AIRS Data

• Measurements from a sensor can have different
  sources of artifacts
• AIRS data has different types of unwanted
  artifacts
• Instrumental noise
• Noisy channels
• Scan-line miscalibration
• Each dot in the image is

15
Instrumental Noise

• Instrumental white noise is unavoidable
• NAPC filtering provides adaptive noise filtering
• NAPC filtering is extensively used in our
  stochastic clearing algorithm and noisy channel
  detector.

Before
After
Histogram of difference
16
Noisy Channels I

• Some channels are consistently noisy
• These channels need to be excluded for further
  analysis and retrieval of physical parameters
• Noisy channel detection is done with NAPC
  filtering such that a channel having 5s even once
  is flagged noisy

Block diagram of noisy channel detector
17
Noisy Channels II

• AIRS science team has their own list of bad
  channels.
• Detection rates for noisy and popping channels

Type of bad channel Number of NASA compiled bad channels Number detected by proposed algorithm detected
High noise 5 5 100
Detector response exhibits unexpected steps (Popping) 2 2 100
Total 7 7 100
18
Scan-line Artifacts I

• Miscalibrated scan-line results
• in stripes
• A simple low-pass filter (in along-track
  direction) in NAPC domain corrects this artifact
  efficiently

Block diagram of removing-stripe algorithm
19
Scan-line Artifacts II

• Results

Original Images
Processed Images
20
Where Are We?

• Problem statement
• Background
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Background and prior work
• Description of stochastic-clearing (SC) algorithm
• Validation of SC algorithm
• Conclusion

21
Background and Prior Work

• What is cloud-clearing?
• Cloudy radiances (or TB) cause inaccurate
  retrieval
• Cloud-cleared radiances radiances which would
  have been observed if FOV contains no clouds
• Prior work on cloud-clearing
• Ignore cloudy FOVs only 5 of AIRS FOVs are
  clear!
• Physical cloud-clearing iterate between
  estimation of physical parameters and calculation
  of observed radiance
• Adjacent-pair clearing use adjacent FOVs which
  have different fractional cloud cover
• Purely spatial processing restore 2-D
  temperature field from sparse clear data

22
Stochastic Clearing (SC)

• How does stochastic clearing (SC) work?
• SC estimates cloud contaminations solely based on
  statistics without using any physical models
• Hyperspectral measurements may contain sufficient
  information about clouds in an obscured manner
• Robust and stable training is necessary
• Nonlinearity is accommodated using stratification
  (sea/land, latitude, day/night), multiplicative
  scan angle correction, etc.
• Advantages of SC approach
• Simple SC does not require physical models
  (retrieval or radiative transfer).
• Fast Based on matrix addition and multiplication

23
Block Diagram of SC Algorithm
D-cloud 2 TBs
1 PC
1
Linear Operator A
Linear Operator B
?
1 PC
N
Select/average FOVs
3x3 AIRS TBs
Less cloudy
Cloudy Test
7
5 microwave ls Land fraction Secant q
More cloudy
?
Linear Operator D
Linear Operator C
Cleared AIRS TBs
N
N 314 channels
24
Operators for SC Algorithm
ECMWF SARTA (v1.05)clear TBs
Trained with gt1000 golfballs
N AIRS TBs
Trained with gt1000 golfballs
Select avg FOVs
N
Operator A
Operator B
Find warmest among 9 pixels
Noise-Adjusted PCs
7
Operators C, D
E S T I M A T O R
L I N E A R

Noise- Adjusted PCs
3
N
Find coldestamong 9 pixels
AIRS D-cloud PCs
-
?TB
AMSU ch. 5,6,8,9,10
5
Warmest/coldest based on 11 4-mm channels
peaking 1-3 km Average 4 warmest pixels for
5-10 km WF, 9 for WF gt10 km

4
N

PC-1
?TBs
secant q
N
Land fraction
AIRS Cleared TBs
25
Clearing Corrections vs ECMWF Ds
Weighting function peak 2.7 km, 2231.5 cm-1
-10 0 10 20 30 40
50 60 70oK
26
AIRS-ECMWF for 827 Channels
Best 22 percent of golfballs (operator
C) Thresholds were 0.8K and 3K Nighttime ocean,
all scan angles Includes all 4- and 15-mm
channels plus one-fifth of the rest
314 good channels
314 good channels

Weighting function peak height (km)
27
Where Are We?

• Problem statement
• Background
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Validation of SC algorithm
• ECMWF
• Physical clearing
• Conclusion

28
ECMWF Data Set Used

• ECMWF profiles are used to simulate cloud-cleared
  TBs via SARTA v1.05 radiative transfer, for all
  scan angles, 314 channels
• Global, 3 days 8/21, 9/3, 10/12/03 (L1B v3.0.8)
• AIRS instrument noise reduced by averaging 1, 4,
  or 9 of the warmest 15-km pixels for weighting
  function peaks 0-5, 5-10, and gt10 km,
  respectively
• Used ?1000 golf balls to regress each of 10
  categories based on day/night, land/sea, and
  latlt40 or 40ltlatlt70

29
RMS Difference (AIRS-ECMWF) for Ocean

• For best 28 sea latlt40º day (best for
  sea)

30
RMS Difference (AIRS-ECMWF) for Land

• For best 28 land latlt40º night (best for
  land)

31
Cloud-cleared AIRS vs. ECMWF (best 28)

• RMS difference (oK) between cloud-cleared AIRS
  and ECMWF/SARTA, 10 different estimators
• RMS gt 0.7 K are boxed
• Excellent agreement for ocean and equatorial
  regions
• Degradation over daytime land near surface

Weighting function peak height (km) Ocean Latlt40 30ltLatlt70 Day Night Day Night Land Latlt40 30ltLatlt70 Day Night All Day Night All
0 1 1 2 4 5 6 7 10 11
0.38 0.4 0.86 0.91 1.68 0.77 1.36 1.48 0.78 1.19 0
.27 0.29 0.54 0.57 0.94 0.38 0.75 0.84 0.44 0.70 0
.28 0.30 0.45 0.45 0.34 0.29 0.33 0.41 0.33 0.39 0
.23 0.27 0.34 0.36 0.25 0.24 0.28 0.34 0.26 0.31 0
.24 0.27 0.33 0.35 0.23 0.25 0.26 0.24 0.28 0.27
32
Global Cloud-Clearing Images (2392.1cm-1)

• SC applied to 8/21/2003 descending orbits (L1B
  v4.0.9)
• 2392.1cm-1, WF peak0.23 km

Observed AIRS
Cloud-cleared (best 78)
33
Global Cloud-Clearing Images (2392.1cm-1)

• CC Residual Spatially high-pass filtered
  version of CC
• WF peaks at 0.23 km

34
Cloud-Clearing Images and Sea Surf. Temperature

• Angle-corrected TB images at window channels
• Clearing works well even if there is no hole
  (clear FOV)

AIRS 2399.9cm-1 near SW Indian Ocean
35
Validation with Physical Clearing Visible vs.
AIRS 8.15-?m Data
Visible Ch 3
1227.7 cm-1
270
265
260
255
250
Granule 91(9/6/02) (solar reflection)
Channel 1284(H2O)
36
Determination of AIRS Cloud-Cleared Brightness
Temperature Baselines
Stochastic CC
Clear Mask
Fitted
Channel 1284 (H2O) (1227.7 cm-1) Granule 91
(9/6/02) Indian Ocean, LAT -26.3, LON 70.2
Clear mask determined from Visible Ch 3, Brown
being clear Polynomial fit 4th order in scan
angle, 3rd order downtrack
37
Stochastic versus Physical Cloud-Cleared8.15-?m
Brightness Temperatures
-Visible Ch 3
Stochastic Clearing
Physical Clearing
Channel 1284 (H2O) (1227.7 cm-1) Granule 91
(9/6/02) Indian Ocean, LAT -26.3, LON 70.2 Filter
subtracts cloud-cleared baseline and convolves
with 3?3 boxcar
38
Correlation Coefficients Visible (v3) vs.
Cloud-Cleared 8.15-?m AIRS TBs
AIRS channel 1284(1227.7 cm-1)(water
vapor)September 6, 2002
Lon -26.3/Lat 70.2 Indian Ocean
Lon -12.5/Lat -106.1 Eastern Southern Pacific
39
Stochastic and Physics-BasedCloud-Cleared Window
Channel TB
Stochastic Clearing
Physics-Based Clearing
Channel 2121 (2399 cm-1), WF peak 200m 9/6/02
North of Bermuda Note that the physical
clearing is most recent version (PGE v4.0.0),
calculated this week.
40
Where Are We?

• Problem statement
• Background
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Validation of SC algorithm
• Conclusion
• Summary / contributions
• Future work

41
Summary

• Anomaly compensation techniques are discussed
  based on signal processing techniques (NAPC and
  ION) and nonlinear estimators
• Anomalies Gaussian instrument noise, noisy
  channels, scan-line miscalibration, and cloud
  contamination
• Stochastic clearing (SC) algorithm tested to be
  successful using different validation schemes
• ECMWF
• Sea surface temperature (SST)
• Physical algorithm

42
Contributions Methodology

• We developed architected nonlinear estimators,
  taking advantage of simplicity of linear
  estimator and robustness of general nonlinear
  estimator
• NAPCs are used to reduce dimension and suppress
  noise, making more robust and stable estimation
• Prior knowledge (either spectral or spatial)
  about anomaly is utilized to meaningfully
  structure a nonlinear estimator

Architected Nonlinear Estimator
Linear Estimator
General Nonlinear Estimator
Dimension reduction
Prior info about anomaly

• May be unstable
• Complex/Slow
• Most powerful

• Stable
• Simple/Fast
• Less powerful

• Stable
• Simple/Fast
• Sufficiently powerful

43
Contributions Stochastic Clearing

• SC algorithm developed based on anomaly
  compensation and nonlinear estimation techniques
• Enjoys excellent agreement with numerical weather
  prediction model (ECMWF)
• Performs superior to physical clearing
• Very fast depends on matrix addition and
  multiplication consists of 664 lines of Matlab
  code, 20 minutes to cloud clear an entire day of
  AIRS data on ordinary PC
• Clearing at extreme scan angles is good
  hole-hunting using high spatial resolution may
  not be essential for cloud-clearing

44
Future Work

• Anomaly compensation theory
• Optimization of combining spectral and spatial
  processing
• More extensive spectral processing techniques
• SC algorithm
• Joint cloud-clearing and retrieval
• Better model for nonlinearities
• Optimum architecture for SC algorithm using
  efficient design-of-experiment approach

45
Where Are We?

• Problem statement
• Background
• Signal characterization and reduction of
  artifacts
• Stochastic cloud-clearing
• Validation of SC algorithm
• Conclusion

Thats all, folks. Any questions?
46
Back-up Slides
47
Global Cloud-Clearing Images (2392.1cm-1)

• Zoom-in images in Southeastern Pacific
• WF peaks at 0.23 km

48
Determination of AIRS Cloud-Cleared Brightness
Temperature Baselines
Stochastic CC
Cloud Mask
Fitted
Channel 1284 (H2O) (1227.7 cm-1) Granule 91
(9/6/02) Indian Ocean, LAT -26.3, LON 70.2
Brown being clear (accepted) Polynomial fit
4th order in scan angle, 3rd order downtrack
49
SC Validation w.r.t. SST

• NCEP sea surface temperature

50
AIRS Stratified Stochastic Cloud Clearing
First-pass results Multiplicative
Scan-angle Stratified results
46 golfballs good 54 rejected
0.8K threshold
(AIRS cleared radiance) (ECMWF/SARTA) (oK)
46 golfballs good 54 rejected
3K threshold
Estimate of cloud-clearing radiance correction
(oK)
51
AIRS Stratified Stochastic Cloud Clearing
First pass, all golfballs, all scan angles Second
pass, multiplicative scan angle Third pass, best
46 golfballs (all scan angle) Third pass, best
46 golfballs using ?lt16?
RMS cleared AIRS radiances vs. ECMWF/SARTA
0.5K
Altitude of weighting function peak (km)
52
AMSU Contributions for Land

• For land 30ºltlatlt70º night

53
Iterative Order and Noise (ION) Estimation

• To apply NAPC, the noise variances need to be
  estimated
• Signal model x Ap Gn, where A is a mixing
  matrix, p is signal of unknown dimension k, G is
  diagonal noise covariance matrix, n is
  unit-variance white Gaussian noise.
• ION iteratively estimates signal order (k),
  mixing matrix (A) and noise variances (G)
• Estimate order (k) using scree plot
• Estimate A and G using Expectation-Maximization
  algorithm

54
Blind Signal Separation

• Observation model x Ap G1/2w
• - A, p, G, w and k (dimension of p) are unknown.
• - A mixing matrix (nk), p source signal
  vector (k1)
• - G diagonal cov matrix (nn), w white
  Gaussian noise vector (n1)
• - Use matrix X, P and W to denote concatenated
  samples of x, p and w
• X AP G1/2 W
• Given X, how to estimate A, G and k.
• Previous signal separation techniques assume
  either k or G is known
• Iterative Order and Noise (ION) estimation
• - First, estimate signal order, k, using
  eigenvector decomposition
• - Estimate A and G using Expectation-Maximization
  algorithm

55
Blind Signal Separation II
X AP G1/2W

• Estimation of signal order, k
• - Selected based on scree plot, sorted
  eigenvalue vs.
• eigenvalue index
• Knee point separates
• signal from noise

56
Blind Signal Separation III
X AP G1/2W

• Expectation-Maximization (EM) algorithm
  iteratively finds maximum likelihood (ML)
  estimate of parameters where model depends on
  hidden (latent) variable
• Expectation step estimate unobserved data (P and
  PTP) using estimate of A and G
• Maximization step compute ML estimate of A and G
  using estimates of P and PTP

57
ION Algorithm Block Diagram
X, optionally normalize rows to zero mean and
unit variance
Set imax, e.g. imax 10
Noise Normalization
Order Estimation
EM Algorithm
Scree Plot
Expectation
Maximization
SVD
Yes
iltimax
No
58
Physics of Radiative Transfer

• Radiative transfer links environmental parameters
  to hyperspectral data
• For a black body, spectral brightness is defined
  as
• For microwave channels (hf ltlt KT) , this is
  linear with temperature

59
Physics of Radiative Transfer

• Upwelling radiation received at a sensor at
  altitude L has four contributions
• Can be rewritten as

60
Physics of Radiative Transfer
AMSU Weighting functions

• Weighting functions for AIRS/AMSU

AIRS Weighting function peaks
61
Physics of Radiative Transfer

• Microwave/IR atmospheric absorption spectrum
• MW Water vapor absorption lines at 22, 183, 325
  GHz, O2 absorption at 118, 368 GHz, etc.
• IR distinct water vapor, O3, CO2 absorptions

Microwave
Infrared