Profile of NPOESS HDF5 Files

1
Profile of NPOESS HDF5 Files

• Kim Tomashosky, Ken Stone, Pat Purcell, Ron
  Andrews
• NPOESS Program
• Raytheon Company
• Aurora, Colorado

2
NPOESS HDF5 General Overview

• Ron Andrews

3
HDF5 Conceptual Diagram
4
HDF5 XML Users Block

• The XML Users Block for NPOESS Data Products
  provides a quick-look into the metadata of the
  associated HDF5 file.
• The XML Users Blocks are defined in the
  following volumes
• CDFCB-X Volume V contains the XML Users Block
  formats for the RDRs, SDRs, TDRs, EDRs, ARPs, and
  IPs
• CDFCB-X Volume VI Ancillary, Auxiliary,
  Reports, and Messages contains the XML Users
  Block formats for the ancillary, auxiliary,
  reports, and messages that are wrapped in HDF5
• The size of the HDF5 XML Users Block will be a
  factor of 512 bytes, with a maximum size which is
  still to be determined
• Provides elements
• N_Processing_Domain
• Mission, Platform, and Instrument Names
• N_Dataset_Type_Tag
• Number_of_Data_Products
• CollectionShortName(s)
• Aggregation Information
• Timestamps (Creation Timestamp, Observation
  Timestamps)
• Percent Missing Data

5
General HDF5 File Structure
6
HDF5 File Root Group Metadata Elements Example
CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
Calendar_Date H_STR X X
Identification_Information.Dataset_Identifier H_STR X X X X X X
Instrument_Short_Name H_STR X X X X X X
Mission_Name H_STR X X X X X X X
N_Creation_Date H_STR X X X X X X X
N_Creation_Time H_STR X X X X X
N_Dataset_Source H_STR X X X X X X X
N_Geo_Ref H_STR X X X X X X X
N_NPOESS_Document_Ref H_STR X X X X
N_Orbit_Position_Predicted (NPOESS only) H_STR X X
N_Subsystem_SW_Version H_STR X X X
N_Update_Date H_STR X
N_Update_Time H_STR X
Platform_Short_Name H_STR X X X X X X
Time_Of_Day H_STR X X
7
HDF5 File Product Group Metadata Elements
Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
Band_ID H_STR X
Band_Measurement_Mode_ID H_STR X X
N_Anc_Type_Tasked H_STR X X
N_Collection_Short_Name H_STR X X X X X X X
N_Dataset_Type_Tag H_STR X X X X X X X
N_Instrument_Flight_SW_Version H_I X X
N_Packet_Type H_STR X
N_Packet_Type_Count H_STR X
N_Processing_Domain H_STR X X X X X X X
N_Spacecraft_SW_Version H_STR X X X
Operational_Mode H_STR X X X
Processing_Level_Identifier H_STR X X X X
8
HDF5 File Product Aggregation Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
AggregateBeginningDate H_STR X X X X X
AggregateBeginningGranuleID H_STR X X X
AggregateBeginningOrbitNumber H_ILLU X X X
AggregateBeginningTime H_STR X X X X
AggregateEndingDate H_STR X X X X
AggregateEndingGranuleID H_STR X X X
AggregateEndingOrbitNumber H_ILLU X X X
AggregateEndingTime H_STR X X X X X
AggregateG-RingLatitude H_F Array X X X
AggregateG-RingLongitude H_F Array X X X
AggregateNumberGranules H_F X X X X
AggregatePercentMissingDataAve H_F X X X X
9
HDF5 File Product Granule Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
Ascending/Descending_Indicator H_C X X
Beginning_Date H_STR X X X X
Beginning_Time H_STR X X X X
Cloud_Cover H_F X
East_Bounding_Coordinate H_STR X X X X
Ending_Date H_STR X X X X
Ending_Time H_STR X X X X
G-Ring_Latitude H_F Array X X X X
G-Ring_Longitude H_F Array X X X X
Input_Dataset_Identifier.Dataset_Identifier H_STR X X X X X X
N_Algorithm_Version H_STR X X X X
N_Anc_Filename H_STR X X
N_Anc_Type H_STR X
N_Aux_Filename H_STR X X X
N_Beginning_Orbit_Number H_ILLU X X X X
N_Beginning_Time_IET H_ILL X X X X X
10
HDF5 File Product Granule Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
N_Columns H_IL X X X
N_Day_Night_Flag H_STR X X X X
N_Deliverable_IP_Filename H_STR X X
N_Ending_Time_IET H_ILL X X X X
N_Graceful_Degradation H_STR X
N_Granule_ID H_STR X X X X
N_Granule_Quality_Value H_F X X X
N_Granule_Version H_STR X X X X X
N_LEOA_Flag H_STR X X X X X
N_Levels H_STR X X
N_LUT_Version H_STR X X
N_Nadir_Latitude_Max H_F X X X
N_Nadir_Latitude_Min H_F X X X
N_Nadir_Longitude_Max H_F X X X
N_Nadir_Longitude_Min H_F X X X
N_Percent_Erroneous_Data H_F X X
11
HDF5 File Product Granule Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
N_Percent_Missing_Data H_F X X X
N_Percent_Not-Applicable_Data H_F X
N_Recent_Maneuver H_F X
N_Reference_ID H_STR X X
N_Rows H_STR X X X X
N_Satellite/Local_Azimuth_Angle_Max H_IL X X X
N_Satellite/Local_Azimuth_Angle_Min H_F X X
N_Satellite/Local_Zenith_Angle_Max H_F X X
N_Satellite/Local_Zenith_Angle_Min H_F X X
N_Solar_Azimuth_Angle_Max H_F X X
N_Solar_Azimuth_Angle_Min H_F X X
N_Solar_Zenith_Angle_Max H_F X X
N_Solar_Zenith_Angle_Min H_F X X
N_Spacecraft_Manuever H_F X X
N_Spacecraft_SW_Version H_STR X X
N_Tile_ID H_STR X X X
12
HDF5 File Product Granule Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
North_Bounding_Coordinate H_I X X X X
Processing_Level_Identifier H_F X X X
South_Bounding_Coordinate H_STR X X X X
West_Bounding_Coordinate H_STR X X X X
13
HDF5 File Spacecraft Diary Group Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
N_Collection_Short_Name H_STR X
N_Dataset_Type_Tag H_STR X
N_Packet_Type H_STR X
N_Packet_Type_Count H_STR X
Processing_Level_Identifier H_STR X
14
HDF5 File Spacecraft Diary Aggregation Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
AggregateBeginningDate H_STR X
AggregateBeginningTime H_STR X
AggregateEndingDate H_STR X
AggregateEndingTime H_STR X
15
HDF5 File Spacecraft Diary Dataset Metadata
Elements Example CDFCB-X Table
Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX
Input_Dataset_Identifier.Dataset_Identifier H_StR X
N_Beginning_Time_IET H_ILL X
N_Ending_Time_IET H_ILL X
N_Support_Data_Version H_STR X
16
HDF5 Conceptual Diagram - Data
17
NPOESS Quality Flags Overview

• The concept is to provide for consistently
  stored, high density, quality information about
  the delivered data simplifying usability while
  maintaining storage efficiency
• Quality flags are qualifications of one or more
  consecutive bits in each byte.
• Quality flag arrays follow the structure of the
  data product
• The size of the arrays are equal to or less than
  the size of the data to which the quality
  information applies (dimensions retain
  correspondence to data product arrays)
• Quality flags will appear in the HDF5 files as n
  number(s) of two or three dimensional, 1-byte
  arrays.
• The number of arrays is dependant on the number
  and size of each data product's quality flag
  definitions
• Quality flags will be packed such that each flag
  is entirely contained within a single byte,
  occasionally resulting in a byte with reserved or
  meaningless bits
• Byte alignment is the same for every quality flag
  array
• First bit is the LSB, bytes are Big-Endian
• Each byte may contain multiple bit-level flags

18
2-Dimensional Array Example
19
3-Dimensional Array Example
20
Detailed NPOESS UML Models

• Ken Stone

21
RDR UML Model
22
Common RDR Layout
23
SDR/TDR UML Model
24
EDR UML Model
25
Ancillary/Auxiliary UML Models
26
NPOESS Sample Data

• Patrick Purcell

27
Product Example UML Model
28
HDF5 View - ImageView
29
Product Profile Example
30
HDF5 API C Reader Sample Code
Note A granule holds a reference to a dataset
region ...
... while an aggregation contains a reference to
an object.
31
NPOESS HDF5 Summary

• The NPOESS Program delivers the official
  deliverable data products (RDR, SDR/TDR,
  EDR/ARP/IP) and dynamic ancillary data and
  auxiliary data in HDF5 Files
• The HDF5 Files have an XML User Block that can be
  accessed without HDF5 tools - provides a
  quick-look into the metadata before opening the
  HDF5 file
• Metadata within the HDF5 files are stored as
  attributes
• There are general UML Models for the NPOESS
  official delivered data that provide a common
  framework
• Official deliverable data products are organized
  by reference objects (aggregations) which contain
  one or more reference regions (granules)
• Although data may be accessed directly through
  the All_Data group, the Data_Products group
  provides integrated access
• Allows the user to access both metadata and data
  through a common HDF5 group
• Metadata is accessed directly by reading the
  Attribute values
• Datasets may be accessed by dereferencing the
  object ID stored in the Data_Products Group for
  the aggregation or granule