Brief Overview: GRIB, HDF, WRF.nc

1
Brief Overview GRIB, HDF, WRF.nc

• NCL supports via addfile function
• GRIB1 and GRIB2
• HDF-SDS Scientific-Data-Set subset of HDF
• HDF-EOS Earth Observing System
• netCDF netCDF-4 with version 5.0.1

• variables imported (read) from grib, hdf,
  hdf-eos
• will have the same consistent structure
• will look/feel like variables read from netCDF
  
• powerful programming concept

2
GRIB1, GRIB2 Gridded Binary

• World Meteorological Org. (WMO) standard
• bit-oriented data format very efficient for
  transmission/archival
• - netCDF (float) 2-4 times larger than
  corresponding GRIB
• two dimensional horizontal grids only
• - each grid has scale/offset for optimal
  precision
• requires external parameter table (s)
• no official read/write API for GRIB makes GRIB
  'problematic'
• - http//www.wmo.ch/web/www/reports/Guide-binary2
  .html
• - ftp//ftp.ncep.noaa.gov/pub/nws/nmc/docs/gribed
  1/

Reanalysis/operational datasets NCEP/ECMWF
use GRIB
3
NCL GRIB

• NCL presents GRIB files/variables as netCDF
  variables
• unpacks 2D grids sorts creates
  multi-dimensional arrays
• creates a variable data object structure
• NCL built-in NCEP/ECMWF parameter tables
• user defined GRIB tables allowed
• (generally) ignore warning messages from NCL for
  GRIB
• http//www.ncl.ucar.edu/Document/Manuals/Ref_Manu
  al/NclFormatSupport.shtml

• GRIB is really a record format and not a file
  format
• records can be placed in any order!!
• NCL must scan entire file
• sort all records to logical grouping by variable
• use information from built-in parameter tables
• creates attributes and corrdinates
• creates values-added variables like 2-dim
  coord arrays

4
NCL GRIB1
ncl 0 gt f addfile ("monthly_1979_1151.grb" ,
"r") ncl 1 gt print (f) like ncdump
h I always do this
Variable f (file variable) filename
monthly_1979_1151.tar path
monthly_1979_1151.tar
note no global atts file global
attributes
grib files have none
dimensions initial_time0 12
lv_ISBL1 17 g0_lat_2 73
g0_lon_3 144 BIG SNIP
ncl 2 gt T f-gtTMP_GDS0_ISBL
printVarSummary(T)
float TMP_GDS0_ISBL ( initial_time0, lv_ISBL1,
g0_lat_2, g0_lon_3 ) center
European Center for Medium-Range Weather
Forecasts long_name Temperature
units K _FillValue
-999 level_indicator 100
grid_number 255 parameter_number
130 forecast_time 0
BIG SNIP
can create netCDF if desired g addfile
(foo.nc,c) g_at_title Grib-to-netCDF g-gtT
TMP_GDS0_ISBL
5
HDF (Hierarchical Data Format)
HDF4 H4 basic features similar to netCDF
- more flexible/features than netCDF but
more complicated

• NCL supports H4 Scientific Data Set (SDS)
  interface only
• - can not access raster, palette,
  Vgroup, Vdata, annotations

• HDF-EOS (EOS Earth Observing System) HE4
• standard HDF with additional conventions,
  metadata
• three geolocation data types
• - grid regular grid, based upon map
  projections
• - point network of irregularly spaced
  locations
• - swath time-sequences of scan-lines, profiles,
  array data
• special interfaces needed to get
  geolocation/temporal info
• http//hdfeos.gsfc.nasa.gov/hdfeos/index.cfm

6
HDF (Hierarchical Data Format)
HDF5 H5 new format (NCL support being
developed) - parallel I/O, no 2GB limit,
data compression - chunking access,
multiple unlimited dimensions
NASA requires H5 in 2004 with launch of Aura
sat. - only occasional exceptions granted

• Unidata/NCSA collaborating to merge netCDF/H5
• will be called netCDF-4 Oct 2006
• H5 will be used as its storage layer

7
HDF4 (Scientific Data Set)
short Avg_Wind_Vel_U ( row, column )
valid_range ltARRAYgt long_name
average east-west comp
of the wind velocity vector
Positive is eastward
units m/s coordsys
Lat/Lon scale_factor 0.01
... snip...
ncl 0gt f addfile ("S3096284.hdf" , "r") ncl 1gt
print (f) Variable f (file variable) filename
S3096284 path S3096284.hdf file
global attributes Producer_Agency NASA
Producer_Institution JPL Sensor_Name
NSCAT ... snip...
Data_Format_Type HDF dimensions row
300 column 720 variables
float row ( row ) long_name
Latitude units degrees_north
float column ( column ) long_name
Longitude units degrees_east
... snip ...
ncl 2 gt u f-gtAvg_Wind_Vel_U ncl 3 gt
printVarSummary (u)
Variable u Type short Total Size 432000 bytes
216000 values Number of Dimensions
2 Dimensions and sizes
row 300 x column
720 Coordinates row -75..74.5
column 0..359.5 Number Of
Attributes 9 ...
snip ...
8
HDF-EOS HE4
ncl 0gt f addfile (BOG99_L2.hdfeos" , "r") ncl
1gt print (f) filename BOG99_L2 file global
attributes" HDFEOSVersion HDFEOS_V2.4
StructMetadata.0 GroupSwathStructure
GroupSWATH_1
SwathNamebog99
GROUPDimension
OBJECTDimension_1
DimensionNameCell_along_Swath"
Size203
END_OBJECTDimension_1
... SNIP other objects ...
END_GROUPDimension
END_GROUPSWATH_1 END_GROUPSwathStructure

Bit 6 1 no ellipsoid
intersection ... snip... byte
Cloud_Mask (byte_segment,cell_Along,cell_Across)
valid_range ltARRAYgt
_FillValue 0 long_name MODIS
Cloud Mask units none
Geolocation_Pointer
External product description bit fields within
byte ordered left to right bit field
description key ------
--------- -------------- 0
cloud mask flag 0 unknown 2,1
quality 00 cloudy
01uncertain
10probably
clear
11confidently clr ...
SNIP... ---------End byte 1 \--------------------
---
dimensions Cell_along_SwathBOG99 203
... snip ... detectors per
250m band 240 // unlimited number of
500m bands 5 // unlimited
detectors per 500m band 20 // unlimited
variables ... snip... byte gflags
(Cell_along_SwathBOG99 ,
1KM_geo_dim ...snip... )
_FillValue 377 Bit 7 (MSB) 1
invalid input dataset
ncl 2 gt cm f-gtCloud_Mask
unpack bits ncl 3 gt flags dim_gbits( ....)
9
WRF Weather Research and Forecast Model
Does not fully conform to common netCDF
standards - variable Time is of type
character not coordinate variable
not numeric monotonic /- - XLAT /
XLONG variables does not use
degrees_north or degrees_east
3-dimensional function of time
- not coordinate variable multi-dim
not monotonic

• global attribute MAP_PROJ tells map projection
• - all required mapping information is
  provided
• f_at_MAP_PROJ

10
WRF netCDF
ncl 0 gt f addfile ("netcdf wrfout_d01_000000.nc"
, "r") ncl 1 gt print (f) like
ncdump h I always do this
dimensions Time UNLIMITED // (25
currently) DateStrLen 19
west_east_stag 91 south_north 81
bottom_top 27 west_east 90
south_north_stag 82
bottom_top_stag 28 soil_layers_stag
4 variables char Times(Time,
DateStrLen) float U(Time, bottom_top,
south_north, west_east_stag)
UFieldType 104 UMemoryOrder
"XYZ" Udescription "x-wind
component" Uunits "m s-1"
BIG SNIP
11
WRF netCDF
float XLAT(Time, south_north, west_east)
XLATFieldType 104
XLATMemoryOrder "XY "
XLATdescription "LATITUDE, SOUTH IS NEGATIVE"
XLATunits "degree"
not udunits conformant
XLATstagger "" float XLONG(Time,
south_north, west_east)
XLONGFieldType 104
XLONGMemoryOrder "XY "
XLONGdescription "LONGITUDE, WEST IS NEGATIVE"
XLONGunits "degree
not udunits conformant
XLONGstagger "" BIG SNIP // global
attributes TITLE " OUTPUT
FROM WRF V1.3 MODEL"
START_DATE "2001-06-11_120000"
WEST-EAST_GRID_DIMENSION 91
SOUTH-NORTH_GRID_DIMENSION 82
BOTTOM-TOP_GRID_DIMENSION 27 snip
CEN_LAT 43.99802f CEN_LON
-98.f TRUELAT1 30.f
TRUELAT2 60.f GMT 12.f
JULYR 2001
JULDAY 162
MAP_PROJ 1