Storing EOS data using HDF5 Archival Information Package

1
Storing EOS data using HDF5 Archival Information
Package

• MuQun Yang (The HDF Group)
• Choonghwan Lee (The HDF Group)
• Ruth Duerr (NSIDC )

2
Definitions of some terms

• CDM(Common Data Model)
• Unification of the data models of OPeNDAP, netCDF
  and HDF5
• CF-Conventions
• Convention of metadata providing spatial
  properties, among climate research community
• CLASS (Comprehensive Large Array-data Stewardship
  System)
• Electronic library of NOAA environmental data

3
Prerequisite

• METS(Metadata Encoding Transmission Standard)
• Standard for encoding structural metadata
• ISO-19115
• International Schema for describing geographic
  information
• File-level Metadata
• Metadata about the individual file or granule
• Dataset-level Metadata
• Metadata that applies to each and every
  granule/file in the whole data set(product)

4
HDF5 Archive Information Package
Data file
HDF5
METS
Metadata file
Primary Schema Extension Schema ltmetsgt ---
ltdmdSecgt----------------ltMODSgt ---ltamdSecgt-------
---------lttechMDgt
--ltrightsMDgt
--ltsourceMDgt ----ltfileGrpgt ----ltstructMapgt
HDF5 AIP Components
http//www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF
5_AIP_White_Paper.pdf
5
NOAA SDS Program
CDM/NetCDF4
ISO-19115
NCDCCLASS
ECS to ISO-19115
HDF5-AIP
NSIDC/ECS Metadata
NetCDF4 / HDF5 Data
METS
ECS to METS
NSIDC/ ECS HDF4-data
H4toH5
NetCDF4/HDF5-data
5
6
Enhanced H4toH5 conversion tool

• Convert HDF-EOS2 data to NetCDF4-compliant HDF5
  data
• Official release (2.0) can be found at
  http//hdfgroup.org/h4toh5/

./h4toh5 eos nc4 input.he2 output.nc4
7
Challenges to do the conversion

• Retrieve geo-location information from HDF-EOS2
  data
• Conform to NetCDF4 data model in the existing
  H4toH5 conversion tool

8
Challenges Handle EOS - Grid

• Grid lacks geolocation fields
• Use predefined projections
• Geographic
• Sinusoidal
• Polar stereographic
• New converter creates geolocation fields
• HDF-EOS2 API GDij2ll()

Data 412 Lon12
Data 48 Lon48
Geographic
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Sinusoidal
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9
Challenges Handle EOS - Swath

• The size of geolocation fields can be different
  from data fields
• New converter has to handle geolocation fields
  correctly

10
Challenges in conforming to NetCDF4
Data field has three columns
Longitude field has two columns

• Follow CF conventions
• Create two variables NewLongitude and
  NewLatitude
• Add to the data field an attribute
  coordinatesNewLongitude NewLatitude
• Keep the original Latitude and Longitude

New longitude has three columns
11
Now some examples to show NetCDF4 files converted
from EOS2
12
A netCDF-4 file converted from EOS2 data at NSIDC
13
A netCDF-4 file converted from EOS2 data at NSIDC
14
Deliverables and future work

• Deliverables
• Enhanced HDF4 to HDF5 conversion tool
• http//hdfgroup.org/h4toh5/
• A validation tool to verify the correctness of
  the conversion
• Will be released soon!
• Potential future work
• 1. Reduce the output file size
• 2. Improve the performance

15
Acknowledgement

• This work was supported under NOAA Scientific
  Stewardship Program grant number NA07OAR4310286.
  Any opinions, findings, and conclusions or
  recommendations expressed in this material are
  those of the author(s) and do not necessarily
  reflect the views of NOAA.