HDF4 Format "Maps" for Long Term Readability

1
HDF4 Format "Maps"for Long Term Readability

• C. Lynnes, GES DISC
• R. Duerr and J. Crider, NSIDC
• M. Yang and P. Cao, The HDF Group

2
In the year 2025...

• A user of HDF-4 data will run into the following
  likely hurdles
• The HDF-4 API and utilities are no longer
  supported...
• ...now that we are at HDF-7
• The archived API binary does not work on today's
  OS's
• ...like Android 3.1
• The source does not compile on the current OS
• ...or is it the compiler version, gcc v. 7.x?
• The HDF spec is too complex to write a simple
  read program...
• ...without re-creating much of the API
• What to do?

3
HDF Mapping Files

• Concept  create text-based "maps" of the HDF-4
  file layouts while we still have a viable HDF-4
  API (i.e., now)
• XML
• Stored separately from, but close to the data
  files
• Includes 
• internal metadata
• variable info
• chunk-level info
• byte offsets and length
• linked blocks
• compression information
• Task funded by ESDIS project
•  The HDF Group, NSIDC and GES DISC

4
Map sample (extract)

•         lthdf4SDS objName"TotalCounts_A"
  objPath"/ascending/Data Fields"
  objID"xid-DFTAG_NDG-5"gt         
  lthdf4Attribute name"_FillValue" ntDesc"16-bit
  signed integer"gt            0 0         
  lt/hdf4Attributegt          lthdf4Datatype
  dtypeClass"INT" dtypeSize"2" byteOrder"BE"
  /gt          lthdf4Dataspace ndims"2"gt          
    180 360          lt/hdf4Dataspacegt         
  lthdf4Datablock nblocks"1"gt           
  lthdf4Block offset"27266625" nbytes"20582"
  compression"coder_typeDEFLATE" /gt         
  lt/hdf4Datablockgt        lt/hdf4SDSgt

5
Status and Future

• Status
• Map creation utility (part of HDF)
• Prototype read programs
• C
• Perl
• Paper in TGRS special issue
• Inventory of HDF-4 data products within EOSDIS
• Possible Future Steps
• Revise XML schema
• Revise map utility and add to HDF baseline
• Implement map creation and storage operationally
• e.g., add to ECS or S4PA metadata files