Ultraspectral Sounder Data Compression

1
Ultraspectral Sounder Data Compression

• Bormin Huang, Allen Huang, Alok Ahuja

Cooperative Institute for Meteorological
Satellite Studies (CIMSS) University of
WisconsinMadison
5th MURI Workshop Madison, WI, June 7-9, 2005
2

• Outline
• Ultraspectral Sounder Data Compression
• Current state-of-the-art Lossless Compression
  Schemes
• 2D JPEG2000
• 3D SPIHT (Set Partitioning In Hierarchical Trees)
• 2D CALIC (Context-based Adaptive Lossless Image
  Codec)
• 2D JPEG-LS
• CIMSSs Data Preprocessing Technique
• Bias-Adjusted Reordering (BAR, 2004)
• CIMSSs New Lossless Compression Schemes
• Predictive Partitioned Vector Quantization (PPVQ,
  2004)
• Fast Precomputed Vector Quantization (FPVQ, 2005)
• Summary

3
(No Transcript)
4
(No Transcript)
5

• Ultraspectral sounder data
• vs.
• Hyperspectral imager data
• Imager data is used in classification, target
  detection and pattern
• recognition. Significant data loss of imager
  data is usually acceptable
• by the human visual system.
• Main criterion of sounder data loss is retrieval
  quality. Retrieval of
• geophysical parameters from observed radiance
  is a mathematically
• ill-posed problem that is sensitive to error
  of data.
• Hence, there is a need for lossless or near
  lossless compression of
• hyperspectral sounder data !!

6
Ultraspectral sounder data for compression studies
AIRS 2378 infrared channels, 135 scan lines x 90
cross-track footprints per granule
7
Wavelet based Schemes

• JPEG2000
• A new ISO/IEC (International Organization for
  Standardization/International Electrotechnical
  Commission) compression standard.
• Successor to the DCT (discrete cosine
  transform)-based JPEG algorithm.

8
3D SPIHT
It uses 3D spatial hierarchical tree
relationship of the wavelet transform
coefficients for efficient compression (Huang et
al. 2003).
Examples of allowable parent-child relations for
2D irregular data
Parent-child interband relationship and locations
for 3D SPIHT coding
9
Predictor-Based Schemes

• 2D CALIC (Context-based Adaptive Lossless Image
  Codec)
• Among the nine proposals in the initial
  ISO/JPEG evaluation in July 1995,
• CALIC was ranked first.
• It is considered the benchmark for lossless
  compression of continuous-tone
• images.

Neighboring pixels used in prediction (Wu et.
al. 1997)
Schematic description of the CALIC encoder
10

• 2D JPEG-LS
• Published in 1999 as a lossless compression
  standard of the ISO/IEC.

Neighborhood of JPEG-LS used in prediction
Schematic description of the JPEG-LS encoder
11
The Bias-Adjusted Reordering (BAR) Scheme (Huang
et al., 2004)
A preprocessing technique for exploring the
correlation among remote disjoint channels to
improve the compression performance of the
existing state-of-the-art schemes.
Given the ith reordered vector , we seek
and to minimize
Then the (i1)-th reordered vector is simply
The optimal value of b is obtained by
which yields
For lossless compression,
is rounded to the nearest integer
, and
the (i1)-th reordered vector becomes
12
PPVQ (Predictive Partitioned Vector Quantization)

• Linear Prediction Each spatial frame is
  estimated from a linear combination of
  neighboring frames.
• Channel Partitioning
• Vector Quantization
• Entropy Coding

13
More results from Serra-Sagrista et al. (IGARSS
2005)
14
Fast Precomputed Vector Quantization
(FPVQ)(Huang et al. 2005)

• Linear Prediction Each channel value is
  estimated from the linear combination of
  neighboring spectral channels
• Bit-depth Partitioning Channels with the same
  bit depth assigned to the same partition
• Vector Quantization with Precomputed Codebooks
  Normalized Gaussian codebooks are used for each
  partition.
• Optimal Bit Allocation An algorithm is presented
  to reduce the expected total number of bits for
  quantization errors.
• Entropy Coding Quantization indices and
  quantization errors are encoded using arithmetic
  coding.

15
Linear Prediction
Each channel is estimated from a linear
combination of np neighboring channels.
or
The prediction coefficients are given by
Prediction error of each channel is close to a
Gaussian distribution with a different standard
deviation.
Examples of Gaussian-like distributions of linear
prediction errors
16
Vector Quantization with Precomputed Codebooks

• Prediction errors of each channel are close to a
  Gaussian distribution with a different standard
  deviation.
• Channels in each partition are represented as a
  linear combination of powers of 2. All 2k
  channels within a partition form a sub-partition.
• Only codebooks with 2m codewords for
  2k-dimensional normalized Gaussian distributions
  are precomputed via the LBG algorithm.
• The actual, data-specific Gaussian codebook is
  the precomputed normalized Gaussian codebook
  scaled by the standard deviation spectrum.

17
Optimal Bit Allocation (Huang et al. 2005)

• Bit allocation algorithms based on marginal
  analysis have been proposed in literature (Riskin
  1991, Cuperman 1993).
• These algorithms may not guarantee an optimal
  solution because they terminate as soon as the
  constraint of their respective minimization
  problems are met, and thus have no chance to move
  further along the hyperplane of the constraint to
  reach a minimum solution.

Bit Allocation Minimization Problem for Lossless
Compression of Ultraspectal Compression
subject to
where
is the expected total bits for the quantization
errors and the quantization indices.
18
New Optimal Bit Allocation Algorithm

• Step 1) Set
• Step 2) Compute the marginal decrement
• Step 3) Find indices for which
  is minimum.
• Step 4) Set
• Step 5) Update
• Step 6) Repeat Steps 3-5 until
• Step 7) Compute the next marginal decrement
• Step 8) Find
  and
• Step 9) If set
  and

19
Example of Optimal Bit Allocation Algorithm
20
Lossless Compression Ratios for AIRS data
21
Summary

• In support of the NOAA/NESDIS GOES-R HES data
  processing studies, we investigated/developed
  lossless compression of 3D hyperspectral sounder
  data using wavelet-based (3D SPIHT, JPEG2000),
  predictor-based schemes (CALIC, JPEG-LS), and
  clustering-based schemes (PPVQ, FPVQ).
• The performance rank in terms of compression
  ratios before our BAR scheme
• JPEG-LS gt 3D SPIHT gt JPEG2000 gt CALIC.
• After our BAR scheme, the compression ratios of
  JPEG-LS, 3D SPIHT, JPEG2000 CALIC are
  significantly improved and they all perform
  almost equally well !
• Our FPVQ PPVQ schemes provide significantly
  higher compression ratios than existing
  start-of-the-art schemes on ultraspectral sounder
  data.

Acknowledgement This research is supported by
NOAA NESDIS OSD under grant NA07EC0676.
22
(No Transcript)