HDF-EOS Overview and Status

1
HDF-EOS Overview and Status

• Larry Klein
• Emergent Information Technologies, Inc.
• September 19, 2000
• larry_at_eos.hitc.com

2
What is HDF-EOS?

• An extension to HDF, built with HDF objects.
• A standard for data structures for remote sensing
  data and in-situ measurements with
• Tightly coupled geolocation information
• Access based on geolocation
• Incorporates ECS metadata attributes.
• Used for many EOS standard products.

3
HDF-EOS File Contents

• Global HDF attributes
• Core Metadata - searchable attributes, copied in
  database
• Archive Metadata - non-searchable attributes
• Any number of HDF objects specifying HDF-EOS
  objects (Grid, Point, Swath)
• Structural metadata - links geolocation to
  science data
• Additional HDF objects.

4
HDF-EOS Grid Structure

• Usage - Data which is organized by regular
  geographic spacing, specified by projection
  parameters.
• Structure
• Any number of 2-D to 8-D data arrays per
  structure, one per data type (e.g. temperature)
• Geolocation information contained in projection
  formula, coupled by structural metadata.
• Any number of Grid structures per file allowed.

5
Grid Data Example
6
Grid Data 1 Km. Digital Elevation Model
South Pole
7
HDF-EOS Swath Structure

• Usage - Data which is organized by time,
  monotonic geolocation, track parameter. Spacing
  can be irregular.
• Structure
• Geolocation information stored explictly in
  Geolocation Field (2-D array)
• Data stored in 2-D or 3-D arrays
• Time stored in 1-D or 2-D array, TAI is standard.
  
• Geolocation/science data connected by structural
  metadata

8
Swath Data Example
9
HDF-EOS Point Structure

• Usage - Data which is specified temporally and/or
  spatially, but with no particular organization.
• Structure
• Tables (Vdatas) used to store science data at a
  particular Lat/Long/Height.
• Up to eight levels of data allowed. Structural
  metadata specifies relationship between levels.

10
Point Data Example
11
HDF-EOS Interface Schematic
User Application
ECS Metadata Interface
HDF-EOS Interface
Grid
Point
Swath
Low Level HDF
File I/O
12
User Interface

• Access - Open, close objects/files
• Definition - Set key features of data sets
• Basic I/O - Read write data
• Index I/O - Link tables (Point only)
• Inquiry - Return information about data
• Subset - Read data from specific region
• Metadata access (SDP toolkit)

13
Current Archive Holdings

• GSFC DAAC MODIS L1, Atmos./Ocean
• 50 TB, 105,000 granules, 300 GB/day
• will be 1.3 TB / day with addition of MODIS L2
  and EOS Aqua data.
• EDC DAAC Landsat/ASTER/MODIS Land
• 20 TB, 106,000 granules, 274 GB/day
• Langely DAAC MISR
• 9 TB, 39,000 granules, 88 GB/day

14
Availability

• HDF library - ftp hdf.ncsa.uiuc.edu
• HDF-EOS V2.6 library (HDF4 based)
• http//newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.h
  tml
• SDP Toolkit library - same location as HDF-EOS
• Platform Support
• DEC, HP, IBM, Sun, SGI, Windows, Linux (soon)
• Language Support - C, C, F77, F90
• HDF4-based library will be supported indefinitely

15
HDF5 - Based Development

• NCSA has introduced HDF5
• A complete rewrite of HDF4, a new user interface.
  
• Simplified data model
• Breaks 2 GB file size limit
• Potential for parallel I/O, multi-threading

16
HDF5 - Based Development HDF-EOS 3.0

• Development overview
• Maintain current HDF-EOS interface to maximum
  extent
• Allow all HDF5 features through interface
• Build HDF-EOS (4 to 5) conversion function
• Data production executable can be built with
  either or both HDF4 and HDF5 - based library
• Support heterogeneous ECS archive (future)

17
Design Schematic
User Application
HDF-EOS Interface Grid, Point, Swath
ECS Metadata Interface

• Common Functions
• Definition, Access, I/O, Inquiry, Subset

HDF4 Interface
HDF5 Interface
Combined HDF4/5 Interface
Build Options
Implemented if required
18
Applications

• H5EOSView - EOSView for HDF5
• Java Browser - Browser for both HDF4 and 5 based
  granules
• HDF-EOS 4 to 5 converter, HDF-EOS 2.x to HDF-EOS
  3.x

19
Schedule

• Jan., 2000 - Beta Version of library
• H5EOSView, Sun, SGI, C support
• Oct. 2000 - Beta2 Version
• Add F77, F90, DEC support
• Local Attributes, Profile Interface,
  Chunking/Tiling
• HDF-EOS 4 to 5 converter and Java Browser

20
Schedule (cont.)

• Jan. 2001 - SCF version
• add HP, IBM, Windows, Linux (maybe a few months
  later), C support
• DAAC version (ECS Drop 6B) - Summer 2001.
• Archives can contain both HDF4 and HDF5 - based
  granules.
• ECS subsystems dont crack granules, but
  regression testing needed.
• 2 GB size limit is an issue.