AIRS Level1B Data Format

1
AIRS Level-1B Data Format

• Evan Manning
• Jet Propulsion Laboratory
• California Institute of Technology

2
AIRS Level-1B Format

• Problem The large size of AIRS Level-1B IR
  products makes it difficult for users to order
  and store as much data as they would like.
• 121 MB per granule
• 29 GB per day.
• 11 TB per year.
• Level-1B V/NIR and Level-2 Cloud-Cleared
  Radiances are each 10 of this.
• Main reason (95 of Level-1B IR) is radiance
  field -- one 32-bit floating-point number per
  channel per FOV.

3
AIRS Level-1B Format

• Investigated solutions (cumulative)
• 1. Scale radiances from 32-bit floating-point to
  16-bit fixed-point
• 2. Rearrange radiances from 135 x 90 spectra to
  2378 images to improve compressibility
• 3. Compress HDF internal or external

4
AIRS Level-1B Format

• 1. Scale radiances from 32-bit floating-point to
  16-bit fixed-point
• Pro
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Harder to use users are responsible for
  rescaling.
• Some "data" lost
• Extreme values that do not fit in range
• Low "noise" bits
• Change from current practice will inconvenience
  current users

5
AIRS Level-1B Format

• 2. Rearrange Radiances from 135 x 90 spectra to
  2378 images.
• Pro
• Improves compressibility
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Harder to use violates HDF-EOS swath format
  conventions, may interfere with standard
  subsetting tools.
• Change from current practice will inconvenience
  current users
• Tradeoff
• Adds no value unless compression is also being
  used.

6
AIRS Level-1B Format

• 3. Compress
• Pro
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Slower access, especially when all data must be
  decompressed to access a subset.
• Harder to use see next slide.

7
AIRS Level-1B Format

• 3. Compress (Cont)

8
AIRS Level-1B Format

• Recommendations
• Implement scaling of AIRS Level-1B IR radiances
  with factor of ½ to ¼ NeNIdeal determined by
  further experiments.
• Investigate using same factors for AIRS Level-2
  cloud-cleared radiances.
• Investigate using similar scaling for AIRS
  Level-1B Vis/NIR radiances.
• Investigate dynamic scaling factors
• Do not reorder radiances or implement compression
  in any AIRS product because the additional size
  savings does not justify the additional
  complexity.
• Encourage users (and GSFC DAAC) to implement
  external compression or internal compression
  (using HDF tool hrepack) where this fits their
  usage profile.

9
Backup Material

• The long form

10
AIRS Level-1B Format

• Problem The large size of AIRS Level-1B IR
  products makes it difficult for users to order
  and store as much data as they would like.
• 121 MB per granule
• 29 GB per day.
• 11 TB per year.
• Level-1B V/NIR and Level-2 Cloud-Cleared
  Radiances are each 10 of this.
• Main reason (95 of Level-1B IR) is radiance
  field -- one 32-bit floating-point number per
  channel per FOV.

11
AIRS Level-1B Format

• Investigated solutions (cumulative)
• 1. Scale radiances from 32-bit floating-point to
  16-bit fixed-point
• 2. Rearrange radiances from 135 x 90 spectra to
  2378 images to improve compressibility
• 3. Compress HDF internal or external

12
AIRS Level-1B Format

• 1. Scale radiances from 32-bit floating-point to
  16-bit fixed-point
• Pro
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Harder to use users are responsible for
  rescaling.
• Some "data" lost
• Extreme values that do not fit in range
• Low "noise" bits
• Change from current practice will inconvenience
  current users

13
AIRS Level-1B Format

• Scale radiances (Details)
• Tradeoffs
• Keeping more low bits reduces noise loss but
  decreases compressibility.
• Dynamic scaling factors increase range and
  compressibility but are more difficult for users.

14
AIRS Level-1B Format

• Proposed scaling Constant per-channel scale
  factor of 1/2 NeNIdeal for each detector, where
  NeNIdeal is determined by a fit to the best
  detectors in each module.

15
AIRS Level-1B Format

• 1. Scale radiances from 32-bit floating-point to
  16-bit fixed-point (Details)
• Experiment Sept 6, 2002 data was processed
  through retrieval 3 ways
• Current Baseline
• After scaling by NeNIdeal/2
• After scaling by NeNIdeal/4
• Results
• When scaling with NeNIdeal/4 yield, Tsurf bias,
  and Tsurf RMS were all indistinguishable from
  baseline.
• When scaling with NeNIdeal/2 bias
• Tsurf bias and RMS were still indistinguishable
  from baseline
• Yield fell by 0.13, but this is within
  experimental uncertainty
• Recommendation
• Scale Level-1B IR radiance product using a
  scaling factor of ½ to ¼ NeNIdeal determined
  by results of further studies

16
AIRS Level-1B Format

• 2. Rearrange Radiances from 135 x 90 spectra to
  2378 images.
• Pro
• Improves compressibility
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Harder to use violates HDF-EOS swath format
  conventions, may interfere with standard
  subsetting tools.
• Change from current practice will inconvenience
  current users
• Tradeoff
• Adds no value unless compression is also being
  used.

17
AIRS Level-1B Format

• 3. Compress
• Pro
• Faster to ftp data.
• Less disk/tape space to store.
• Con
• Slower access, especially when all data must be
  decompressed to access a subset.
• Harder to use see next slide.

18
AIRS Level-1B Format

• 3. Compress (Details)
• Tradeoffs
• Added time
• 30 sec to compress a 6-minute granule.
• 15 sec to decompress a 6-minute granule.
• External compression puts responsibility on users
  and requires decompression for even trivial data
  access.
• The most aggressive compression utilities (bzip2)
  are not universally supported.
• HDF-EOS supports internal compression which is
  transparent to users, but not very aggressive.
• New HDF-EOS 5 supports more aggressive internal
  compression, but would be an even greater effort
  to implement and even harder for existing users
  to adjust to.
• HDF hrepack tool may allow users to apply or
  reverse HDF internal compression to product files
  like external compression.

19
AIRS Level-1B Format

• 3. Compress (Details - Cont)
• Experiment
• Scale one granule by NeNIdeal/2.
• With and without reordering.
• Best HDF internal compression is skipping
  Huffman.
• Best tested external compression is bzip2 9.

20
AIRS Level-1B Format

• 3. Compress (Cont)

21
AIRS Level-1B Format

• 3. Compress (Details - Cont)
• Notes on Experiment
• Greater compression would be realized with
  dynamic scaling.
• Lower compression would be realized with smaller
  scaling bin size.
• Sample size for this experiment is small (1
  granule), and exact ratios vary with scene
  characteristics, so all numbers are approximate.
• Overall compression ratios are limited by
  uncompressed 5 overhead of metadata and
  other fields (geolocation, QA, etc.).

22
AIRS Level-1B Format

• Recommendations
• Implement scaling of AIRS Level-1B IR radiances
  with factor of ½ to ¼ NeNIdeal determined by
  further experiments.
• Investigate using same factors for AIRS Level-2
  cloud-cleared radiances.
• Investigate using similar scaling for AIRS
  Level-1B Vis/NIR radiances.
• Do not reorder radiances or implement compression
  in any AIRS product because the additional size
  savings does not justify the additional
  complexity.
• Encourage users to implement external compression
  or internal compression (using HDF tool hrepack)
  where this fits their usage profile.

23
Deep Backup Material

• Ideal NeDTs for each module

24
AIRS Level-1B Format

• Ideal (fit) NeDT for Modules M1 M2

25
AIRS Level-1B Format

• Ideal (fit) NeDT for Modules M3 M4

26
AIRS Level-1B Format

• Ideal (fit) NeDT for Modules M5 M8

27
AIRS Level-1B Format

• Ideal (fit) NeDT for Modules M9 M12