HDFEOS 3'0 Functional and Structural Design

1
HDF-EOS 3.0 Functional and Structural Design
David Wynne/Alex Muslimov Emergent Information
Technologies, Inc. September 19,
2000 davidw_at_eos.hitc.com/amuslimo_at_eos.hitc.com
2
HDF-EOS 3.0

• Based on HDF5, with input from NCSA with respect
  to Design, Implementation and Method
• Working with EOS AURA teams to Refine Design
• Primary Goal of Design -- minimal impact to
  Heritage Code

3
Current HDF-EOS 3.0 Functionality

• Basic File I/O
• FillValues
• Compression
• Chunking/Tiling
• Swath Interface
• Grid Interface
• FORTRAN Wrappers
• Profile Interface

• Global, Group Local Attributes
• Extendable DataSets
• Test Drivers
• Users Guide
• Subsetting
• SUN, SGI DEC

4
Functional Specification

• Swath File Access
• SWopen, SWcreate, SWattach, SWdetach and SWclose
• Dimension/ Mapping Definition
• SWdefdim, SWdefdimmap and SWdefidxmap
• Field Definition/Compression/Chunking
• SWdefgeofield, SWdefdatafield, SWdefcomp and
• SWdefcomchunk
• Field I/O, Inquiry and Info
• SWwritefield, SWreadfield, SWinqdatafields,
  SWinqgeofield
• and SWfieldinfo
• Function interface has changed between HDF-EOS
  2.x and 3.0
• Function names italicized represent new
  functionality

5
Functional Specification, cont.

• Attribute I/O, Inquiry and Info
• SWwriteattr, SWreadattr, SWinqattrs, SWattrinfo
• SWwritegrpattr, SWreadgrpattr, SWinqgrpattrs,
  SWgrpattrinfo
• SWwritelocattr, SWreadlocattr, SWinqlocattrs and
  SWlocattrinfo
• General Inquiry and Info
• SWinqswath, SWnentries, SWinqdims, SWdiminfo,
• SWinqmaps, SWmapinfo, SWinqidxmaps and
  SWidxmapinfo
• Subsetting
• SWdefboxregion, SWdeftimeperiod, SWregioninfo,
• SWperiodinfo, SWextractregion and SWextractperiod
• Profile
• PRdefine, PRread, PRwrite, PRinquire and PRinfo
• Function names italicized represent new
  functionality

6
Functional Specification, cont.

• HDF-EOS 2.6
• SWdefdatafield(swathID, fieldname, dimlist,
  numtype, merge)
• HDF-EOS 3.0
• SWdefdatafield(swathID, fieldname, dimlist,
  maxdimlist, numtype, merge)
• Maxdimlist
• New HDF5 functionality passed through. This
  allows the user to set an upper limit to the size
  of the dataset.

7
Top Level of HDF-EOS 3.0 File
Root -- /
HDFEOS
POINTS
SWATHS
GRIDS
HDFEOS INFORMATION
Miscellaneous
Proposed Group, User Input
Group
8
Top Level of HDF-EOS 3.0 File
HDFEOS
HDFEOS INFORMATION
HDFEOSVersion
StructMetadata.0
Data Set
Attribute
Group
9
Swath Structure
SwathName
Global Attribute ltSwathNamegtltAttrNamegt
Group Attribute ltDataFieldsgtltAttrNamegt
Data Fields
Geolocation Fields
Profile Fields
Local Attribute ltFieldNamegtltAttrNamegt
Profile Field1
Profile Fieldn
Longitude
Latitude
Data Field1
Data Fieldn
CoLatitude
Time
Each Data Field can have Attributes and/or
Dimension Scales
Dimension Scale
Shaded Objects are implemented in a fixed way so
the user doesnt have direct access via the
interface
DataField1DIMENSIONLIST
Proposed Attribute, User Input
Message Attribute ltSwathNamegtltAttrNamegt
Data Set
Attribute
Group
10
Profile Structure
SwathName
Global Attribute ltSwathNamegtltAttrNamegt
Group Attribute ltDataFieldsgtltAttrNamegt
Profile Fields
Geolocation Fields
Local Attribute ltFieldNamegtltAttrNamegt
Profile Field1
Profile Fieldn
Longitude
Latitude
Each Data Field can have Attributes and/or
Dimension Scales
Dimension Scale
CoLatitude
Time
DataField1DIMENSIONLIST
Message Attribute ltSwathNamegtltAttrNamegt
Proposed Attribute, User Input
Data Set
Attribute
Group
11
Grid Structure
Global Attribute ltGridNamegtltAttrNamegt
GridName
_BLKSOM
Group Attribute ltDataFieldsgtltAttrNamegt
Data Fields
Local Attribute ltFieldNamegtltAttrNamegt
Data Field1
Data Fieldn
Message Attribute ltGridNamegtltAttrNamegt
Each Data Field can have Attributes and/or
Dimension Scales
Dimension Scale
Proposed Attribute, User Input
Shaded Objects are implemented in a fixed way so
the user doesnt have direct access
DataField1DIMENSIONLIST
Data Set
Attribute
Group
12
Point Structure
PointName
Global Attribute ltPointNamegtltAttrNamegt
_LevelWritten
Group Attribute ltDataFieldsgtltAttrNamegt
Linkage
Data
Object Reference
FWD POINT
Level1 Name
Local Attribute ltLevelNamegtltAttrNamegt
BCK POINT
Leveln Name
Object Reference
Message Attribute ltPointNamegtltAttrNamegt
Shaded Objects are implemented in a fixed way so
the user doesnt have direct access
Proposed Attribute, User Input
Data Set
Group
Attribute
13
Issues

• Maximum size of variable length and compound data
  types
• Chunking with variable length and compound data
  types
• Only deflation compression method implemented.
• File size limit in ECS - Is two Gbytes enough?