Profile of NPOESS HDF5 Files

1
Profile of NPOESS HDF5 Files

• William Johnsen, Ph.D., Chad Johnson
• NPOESS Program
• Raytheon Company
• Aurora, Colorado

2
General HDF5 File Structure for NPP Data Products
Root Group
ltltRootgtgt
Attributes are
NPOESS General HDF_File_Root
here.
The All_Data group is
xxx_Data contains the sensor
The xxx_Product_Data
populated with nested
data for a given NPOESS
contains a collection of
A root group may
xxx_Product_Data groups.
data product
product datasets
have one or more
product groups.
Product groups
ltltH_Ggtgt
ltltH_Ggtgt
ltltH_DSgtgt
ltltH_Ggtgt
contain datasets
HDF_xxx_Product_Data
HDF_All_Data
xxx_Data
HDF_xxx_Product
for aggregations
(from NPOESS General HDF_File_Root)
...
)
and individual
granules.
1
ltltH_DSgtgt
ltltH_DSgtgt
The Product Granule
1...
contains a regional
HDF_xxx_Product_Aggregation
HDF_xxx_Product_Granule_
reference (H_RR) to
(from HDF_xxx_Product)
(from HDF_xxx_Product)
the data.
AggregateDataset H_RO
Dataset H_RR
xxx is the product type,
The Product
for example RDR,
Aggregation contains a
SDR, EDR, etc.
regional object
is a number
(H_RO) to the data.
corresponding to the
xxx is the product type,
The dataset is a
granule number
for example RDR,
regional
within the aggregated
SDR, EDR, etc.
reference (H_RO)
dataset.
is a number
to the data.
corresponding to the
granule number within
ltltH_RR Arraygtgt
the aggregated
The dataset is a
ltltH_RO Arraygtgt
DatasetArray
dataset.
regional
AggregateDatasetArray
(from HDF_xxx_Product_Granule_)
...
)
reference (H_RR)
(from HDF_xxx_Product_Aggregation)
...
)
to the data.
3
General EDR File Structure relationships
ltltRootgtgt
HDF_File_Root
1
1
ltltH_Ggtgt
1
1
ltltH_Ggtgt
1
1
HDF_All_Data
HDF_EDR_Product Group
(from HDF_File_Root)
1
1
1..
1..
ltltH_Ggtgt
HDF_xxx_Product_Data
1
1
1..
1..
1..n
1..n
1
1
ltltH_DSgtgt
ltltH_DSgtgt
ltltH_DSgtgt
xxx_Data
HDF_EDR_Product_Granule_n Dataset
HDF_EDR_Product_Aggregation Dataset
(from HDF_EDR_Product Group)
(from HDF_EDR_Product Group)
A H_DC is attached to each
H_DS item. See specific
EDR diagrams for details.
Note All NPP EDR HDF5 files contain the
HDF_All_Data Group thus will not be depicted in
the following diagrams.
4
VIIRS Aerosol Optical Thickness
ltltH_DSgtgt
HDF_EDR_Product_Aggregation Dataset
(from HDF_EDR_Product Group)
ltltNPOESSgtgt AggregateBeginningDate H_CU
ltltH_RO Arraygtgt
ltltNPOESSgtgt AggregateBeginningGranuleID H_IL
(EDR VIIRS Aerosol Optical Thickness Aggregation)
ltltRootgtgt
ltltNPOESSgtgt AggregateBeginningOrbitNumber H_IL
(from HDF_EDR_Product_Aggregation Dataset)
HDF_File_Root
ltltNPOESSgtgt AggregateBeginningTime H_CU
AOT_Land Ref H_RO
Mission_Name H_STR
ltltNPOESSgtgt AggregateCreationDate H_STR
AOT_Water Ref H_RO
ltltNPOESSgtgt AggregateCreationTime H_STR
N_Quasi-Static_Metadata_Reference
1
1
Quality Flags Ref H_RO
Identifier_Information.Dataset_Indentifier H_STR
ltltNPOESSgtgt AggregateEndingDate H_CU
ltltNPOESSgtgt AggregateEndingGranuleID H_IL
ltltNPOESSgtgt AggregateEndingOrbitNumber H_IL
1
1
ltltNPOESSgtgt AggregateEndingTime H_CU
ltltNPOESSgtgt AggregateG-RingLatitude H_F Array
ltltNPOESSgtgt AggregateG-RingLongitude H_F Array
ltltNPOESSgtgt AggregatePercentMissingDataAve H_F
1
1
ltltH_DSgtgt
HDF_EDR_Product_Granule_n Dataset
1
1
(from HDF_EDR_Product Group)
ltltH_Ggtgt
Ascending/Decending_Indicator H_STR
HDF_EDR_Product Group
Beginning_Date H_CU
(from HDF_File_Root)
Beginning_Time H_CU
Band_Measurement_Mode_ID H_STR
Cloud_Cover H_I
Instrument_Short_Name H_STR
East_Bounding_Coordinate H_F
N_Dataset_Type_Tag H_STR
Ending_Date H_CU
N_Instrument_Flight_SW_Version H_STR
Ending_Time H_CU
N_Spacecraft_SW_Version H_STR
G_Ring_Latitude H_F Array
N_Subsystem_SW_Version H_STR
G-Ring_Longitude H_F Array
Platform_Short_Name H_STR
Input_Dataset_Identifier.Dataset_Identifier
H_STR
Processing_Level_Identifier H_STR
N_Algorithm_Version H_STR
N_Anc_Type_Used N_STR
N_Automatic_Quality_Parameter_Value H_F
N_Automatic_Quality_Test_Explanation H_STR
N_Beginning_Orbit_Number H_UI
1..n
1..n
N_Beginning_Time_IET H_LL
ltltH_RR Arraygtgt
N_Creation_Date H_STR
(EDR VIIRS Aerosol Optical Thickness)
N_Creation_Time H_STR
(from HDF_EDR_Product_Granule_n Dataset)
N_Day_Night_Flag H_F
AOT_Land Ref H_RR
N_Degraded_Mode H_STR
AOT_Water Ref H_RR
1
1
N_EDR_Field H_F
Quality Flags Ref H_RR
N_EDR_Scale_Factor H_F
N_Ending_Time_IET H_LL
N_Graceful_Degradation H_STR
N_Granule_ID H_I
These Arrays are different for each product.
Subsequent slides show only these Arrays for each
data product.
N_Granule_Version H_STR
N_LEO_Flag H_STR
N_Nadir_Latitude_Max H_F
N_Nadir_Latitude_Min H_F
N_Nadir_Longitude_Max H_F
N_Nadir_Longitude_Min H_F
N_Percent_Missing_Data H_F
N_Solar_Zenith_Angle_Max H_F
N_Solar_Zenith_Angle_Min H_F
N_Update_Date H_STR
N_Update_Time H_STR
North_Bounding_Coordinate H_F
Operational_Mode H_STR
South_Bounding_Cooridinate H_F
West_Bounding_Coordinate H_F
5
HDFView Snapshot
6
xDR Example Data - HDFView
xDR VIIRS SST Num Granules 50 Rows
128,000 Columns 3,200 Dimensions 100 x 1000
km2 Data Source Simulated Propagating Sine Wave,
4 phase, 1.51 phase shift
xDR VIIRS Cloud Mask Scan All Ocean Num
Granules 12 Rows 3,072 Columns
3,200 Dimensions 100 x 1000 km2 Data Source
Simulated Propagating Sine Wave, 4 phase, 1.51
phase shift
xDR VIIRS NCC Imagery DNB Num Granules 1 Rows
256 Columns 4,064 Dimensions 100 x 1000
km2 Data Source MODIS plus granule edge fill
data
7
VIIRS Aerosol Particle Size Parameter
VIIRS Cloud EffectiveParticle Size
VIIRS Cloud Cover/Layer
VIIRS Cloud Height
8
VIIRS Cloud Optical Thickness
VIIRS Cloud Top Pressure
VIIRS Cloud Top Temperature
VIIRS Cloud Top Height
9
VIIRS Cloud Mask IP
VIIRS Ice Surface Temperature
VIIRS I1-Channel Imagery
VIIRS Imagery NCC
10
VIIRS Land Surface Temperature
VIIRS Ocean Color/Chlorophyll
VIIRS Sea Ice Characterization
VIIRS Net Heat Flux
11
VIIRS Sea Surface Temperature
VIIRS Soil Moisture
VIIRS Surface Albedo
VIIRS Snow Cover/Depth
12
VIIRS Surface Type
VIIRS Vegetation Index
VIIRS Suspended Matter
13
General RDR File Structure relationships
ltltRootgtgt
HDF_File_Root
ltltH_Ggtgt
1
1
ltltH_Ggtgt
HDF_All_Data
HDF_RDR_Product
1
1
1..
1..
ltltH_Ggtgt
HDF_xxx_Product_Data
ltltH_DSgtgt
ltltH_DSgtgt
1
1
RDR_Product_Granule_
1
1
ltltH_Ggtgt
ltltH_Ggtgt
RDR_Product_Aggregation
1
1
HDF_RDR_SpacecraftDiary
HDF_AUX_SensorCalibrationCoefficient
1..
1..
ltltH_DSgtgt
xxx_Data
ltltH_DSgtgt
HDF_RDR_SpacecraftDiary_Dataset_
ltltH_DSgtgt
HDF_AUX_SensorCalibrationCoefficient_Dataset_
1
1
ltltH_DSgtgt
HDF_RDR_SpacecraftDiary_Aggregation
ltltH_DSgtgt
HDF_AUX_SensorCalibrationCoefficient_Aggregation
Note All NPP RDR HDF5 files contain the
HDF_All_Data Group thus will not be depicted in
the following diagrams.
14
RDR
ltltH_DSgtgt
RDR_Product_Aggregation
(from HDF_RDR_Product)
ltltNPOESSgtgt AggregateBeginningDate H_CU
ltltNPOESSgtgt AggregateBeginningOrbitNumber H_ILLU
ltltNPOESSgtgt AggregateBeginningTime H_CU
ltltNPOESSgtgt AggregateBeginningGranuleID H_I
ltltNPOESSgtgt AggregateCreationDate H_STR
ltltNPOESSgtgt AggregateCreationTime H_STR
ltltNPOESSgtgt AggregateEndingGranuleID H_I
ltltNPOESSgtgt AggregateEndingDate H_CU
ltltNPOESSgtgt AggregateEndingOrbitNumber H_ILLU
ltltNPOESSgtgt AggregateEndingTime H_CU
ltltNPOESSgtgt AggregatePercentMissingDataAve H_F
ltltHDFgtgt AggregateDataset H_RO
ltltH_DSgtgt
RDR_Product_Granule_
(from HDF_RDR_Product)
Beginning_Date H_CU
Beginning_Time H_CU
Ending_Date H_CU
Ending_Time H_CU
N_Beginning_Orbit_Number H_ILLU NO
N_Beginning_Time_IET H_I
N_Creation_Date H_STR
N_Creation_Time H_STR
N_Ending_Time_IET H_I
ltltH_Ggtgt
N_Granule_ID H_I
HDF_RDR_Product
N_Granule_Version H_STR
(from HDF_File_Root)
N_LEO_Flag H_STR
Instrument_Short_Name H_STR
N_Packet_Type H_STR
N_Dataset_Type_Tag H_STR
N_Packet_Type_Count H_UI
N_Instrument_Flight_SW_Version H_STR
N_Percent_Missing_Data H_I
N_Spacecraft_SW_Version H_STR
N_Update_Date H_STR
ltltH_DSgtgt
N_Subsystem_SW_Version H_STR
N_Update_Time H_STR
HDF_RDR_SpacecraftDiary_Dataset_
Platform_Short_Name H_STR
Operational_Mode H_STR
Processing_Level_Identifier N_STR
(from HDF_RDR_SpacecraftDiary)
ltltHDFgtgt Dataset H_RR
N_Dataset_Source H_STR
Beginning_Date H_CU
Beginning_Time H_CU
N_Processing_Domain H_STR
N_Granules H_I
Ending_Date H_CU
Identifier_Information.Dataset_Identifier H_STR
Ending_Time H_CU
N_Beginning_Orbit_Number H_ILLU
1
1
N_Beginning_Time_IET H_I
N_Creation_Date H_STR
N_Creation_Time H_STR
N_Ending_Time_IET H_I
N_Granule_ID H_I
N_Granule_Version H_STR
ltltH_Ggtgt
N_LEO_Flag H_STR
HDF_RDR_SpacecraftDiary
N_Packet_Type H_STR
(from HDF_File_Root)
N_Packet_Type_Count H_UI
Instrument_Short_Name H_STR
N_Percent_Missing_Data H_I
ltltRootgtgt
Identifier_Information.Dataset_Indentifier H_STR
N_Update_Date H_STR
N_Dataset_Type_Tag H_STR
HDF_File_Root
N_Update_Time H_STR
N_Instrument_Flight_SW_Version H_STR
Mission_Name H_STR
Operational_Mode H_STR
N_Spacecraft_SW_Version H_STR
ltltHDFgtgt Dataset H_RR
N_Quasi-Static_Metadata_Reference
1
1
N_Subsystem_SW_Version H_STR
Identifier_Information.Dataset_Indentifier H_STR
Platform_Short_Name H_STR
Processing_Level_Identifier N_STR
ltltH_DSgtgt
N_Dataset_Source H_STR
HDF_RDR_SpacecraftDiary_Aggregation
N_Processing_Domain H_STR
(from HDF_RDR_SpacecraftDiary)
ltltNPOESSgtgt AggregateBeginningDate H_CU
ltltNPOESSgtgt AggregateBeginningOrbitNumber H_ILLU
1
1
ltltNPOESSgtgt AggregateBeginningTime H_CU
1
1
ltltNPOESSgtgt AggregateBeginningGranuleID H_I
ltltNPOESSgtgt AggregateCreationDate H_CU
ltltH_Ggtgt
ltltNPOESSgtgt AggregateCreationTime H_CU
HDF_AUX_SensorCalibrationCoefficient
ltltNPOESSgtgt AggregateEndingGranuleID H_I
(from HDF_File_Root)
ltltNPOESSgtgt AggregateEndingDate H_CU
N_Dataset_Source H_STR
ltltNPOESSgtgt AggregateEndingOrbitNumber H_ILLU
N_Dataset_Type_Tag H_STR
ltltNPOESSgtgt AggregateEndingTime H_CU
N_Processing_Domain H_STR
ltltNPOESSgtgt AggregatePercentMissingDataAve H_F
N_Spacecraft_SW_Version H_STR
ltltHDFgtgt AggregateDataset H_RO
N_Subsystem_SW_Version H_STR
Identifier_Information.Dataset_Indentifier H_STR
ltltH_DSgtgt
HDF_AUX_SensorCalibrationCoefficient_Aggregation
(from HDF_AUX_SensorCalibrationCoefficient)
ltltNPOESSgtgt AggregateBeginningDate H_CU
ltltNPOESSgtgt AggregateBeginningTime H_CU
ltltNPOESSgtgt AggregateEndingDate H_CU
ltltNPOESSgtgt AggregateEndingTime H_CU
ltltHDFgtgt AggregateDataset H_RO
ltltH_DSgtgt
HDF_AUX_SensorCalibrationCoefficient_Dataset_
(from HDF_AUX_SensorCalibrationCoefficient)
Ancillary_Dataset_Identifier.Dataset_Identifier
H_STR
Calendar_Date
Input_Dataset_Identifier.Dataset_Identifier
H_STR
N_Beginning_Time_IET H_LL
N_Creation_Date H_STR
N_Creation_Time H_STR
N_Ending_Time_IET H_LL
N_LEO_Flag H_STR
N_Update_Date H_STR
N_Update_Time H_STR
Platform_Short_Name H_STR
Time_of_Day H_STR
ltltHDFgtgt Dataset H_RR
15
Ancillary / Auxiliary Data Products
ltltRootgtgt
ltltRootgtgt
ANC_HDF_File_Root Group
AUX_HDF_File_Root group
Ancillary_Dataset_Identifier.Dataset_Identifier
H_STR
Ancillary_Dataset_Identifier.Dataset_Identifier
H_STR
Identification_Information.Citation.Citation_Infor
mation.Online-Linkage H_STR
Identification_Information.Citation.Citation_Infor
mation.Online-Linkage H_STR
Identifier_Information.Dataset_Indentifier H_STR
Identifier_Information.Dataset_Indentifier H_STR
Input_Dataset_Identifier.Dataset_Identifier
H_STR
Input_Dataset_Identifier.Dataset_Identifier
H_STR
Mission_Name H_STR
Instrument_Short_Name H_STR
N_Anc_Type H_STR
Mission_Name H_STR
N_Creation_Date H_STR
N_Anc_Type H_STR
N_Creation_Time H_STR
N_Beginning_Time_IET H_LL
N_Dataset_Source H_STR
N_Creation_Date H_STR
N_Dataset_Type_Tag H_STR
N_Creation_Time H_STR
N_Processing_Domain H_STR
N_Dataset_Source H_STR
N_Update_Date H_STR
N_Dataset_Type_Tag H_STR
N_Update_Time H_STR
N_Ending_Time_IET H_LL
N_LEO_Flag H_STR
N_Processing_Domain H_STR
1
1
N_Spacecraft_SW_Version H_STR
N_Subsystem_SW_Version H_STR
N_Update_Date H_STR
N_Update_Time H_STR
Platform_Short_Name H_STR
1
1
1
1
ltltH_Ggtgt
1
1
HDF_All_Data
ltltH_Ggtgt
AncillaryData
1
1
1
1
(from ANC_HDF_File_Root Group)
1
1
ltltH_Ggtgt
Calander_Date H_STR
HDF_All_Data
ltltH_Ggtgt
1..
1..
Time_of_Day H_STR
AuxiliaryData
ltltH_Ggtgt
(from AUX_HDF_File_Root group)
1
1
HDF_xxx_Product_Data
Calendar_Date
N_Granule_Version H_STR
1..
1..
1
1
Time_of_Day H_STR
ltltH_Ggtgt
HDF_xxx_Product_Data
1..
1..
ltltH_DSgtgt
1
1
xxx_Data
1..
1..
ltltH_DSgtgt
xxx_Data
16
General SDR File Structure relationships
ltltRootgtgt
HDF_File_Root
1
1
ltltH_Ggtgt
1
1
ltltH_Ggtgt
1
1
HDF_All_Data
HDF_SDR_Product Group
1
1
1
1
1
1
1..
1..
ltltH_Ggtgt
HDF_xxx_Product_Data
1
1
1..
1..
1
1
ltltH_DSgtgt
ltltH_DSgtgt
HDF_SDR_Product_Granule_n Dataset
HDF_SDR_Product_Aggregation Dataset
1..
1..
ltltH_DSgtgt
xxx_Data
A H_DC is attached to each
H_DS item. See specific
EDR diagrams for details.
Note All NPP SDR HDF5 files contain the
HDF_All_Data Group thus will not be depicted in
the following diagrams.
17
HDF5 file structure Example CDFCB-X Table for
RDR Product Granule Group (primary data)
Data types are explained on chart 15-16
18
HDF5 file structure Example CDFCB-X Table for
RDR Auxiliary Data Calibration Coefficients Group
(secondary data)
Data types are explained on chart 15-16
19
HDF5 file structure HDF5 data type to CDFCB-X
cross-reference
20
HDF5 file structure HDF5 attribute to CDFCB-X
cross-reference