TIGGE, GRIB to NetCDF converter

1
TIGGE, GRIB to NetCDF converter

• Doug Schuster
• (NCAR/ECMWF)

2
Topic Outline

• Introduction
• Ensemble NetCDF File Structure
• Converter Highlights
• Challenges
• Future Plans

3
GRIB to NetCDF Conversion
GRIB File
NetCDF File
Metadata
t, EGRR, 1
d, ECMF, 2
Gather metadata and message locations
t (1,2,3,4)
t, ECMF, 2
t, EGRR, 2
Create NetCDF file structure
d, EGRR, 1
d (1,2,3,4)
t, ECMF, 1
Populate NetCDF parameter arrays (1,2,3,4)
represents ensemble member id (Realization)
d, EGRR, 2
d, ECMF, 1
4
Ensemble NetCDF File Structure

• NetCDF File format
• Based on available CF conventions
• File organization built according to Doblas-Reyes
  (ENSEMBLES project) proposed NetCDF file
  structure
• Provides grid/ensemble specific metadata for each
  member
• Data Provider
• Forecast type (perturbed, control, deterministic)
• Allows for multiple combinations of
  initialization times and forecast periods within
  one file.
• Pairs of initialization and forecast step

5
Ensemble NetCDF File Structure

• NetCDF Parameter structure (5 dimensions)
• Reftime
• Realization (Ensemble member id)
• Level
• Latitude
• Longitude
• Coordinate variables are use to describe
• Realization
• Provides metadata associated with each ensemble
  grid.
• Reftime
• Allows for multiple initialization times and
  forecast periods to be contained within one file

6
Ensemble NetCDF File Structure

• Coordinate variables for Realization
• Provide ensemble member metadata
• Institution
• Forecast Type

Realization 1 2 3 4 Param(Realization) Instit
ution(Realization) ECMF ECMF EGRR EGRR ForecastTy
pe(Realization) cf pf cf pf
Grid 1
Grid 2
Grid 3
Grid 4
-At fixed Reftime and Level
7
Ensemble NetCDF File Structure

• Coordinate variable for Reftime
• Provide Forecast Times for each Initialization
• Leadtime

Reftime 20071012,12z 20071013,00z 20071013,00z P
aram(Reftime) Leadtime(Reftime) 24 12 18
Grid 1
Grid 2
Grid 3
-At fixed Realization and Level
8
Ensemble NetCDF File Structure

• CDL Examples (reftime and coordinate variable)
• int reftime(reftime)
• reftimedata_type "long"
• reftimeunits "hours since 1950-01-01
  000000"
• reftimestandard_name "forecast_reference_
  time"
• reftimelong_name "Time of model
  initialization"
• int leadtime(reftime)
• leadtimedata_type "int"
• leadtimeunits "hours"
• leadtimestandard_name "forecast_period"
  
• leadtimelong_name "hours since
  forecast_reference_time"

9
Ensemble NetCDF File Structure

• CDL Examples (realization and coordinate
  variables)
• int realization(realization)
• realizationdata_type"int"
• realizationunits "1"
• realizationstandard_name "realization"
• realizationlong_name "Number of the
  simulation in the
• 
  ensemble"
• char forecast_type(realization, string2)
• forecast_typedata_type "char"
• forecast_typestandard_name
  "forecast_type"
• forecast_typelong_name Forecast type"
  
• char institution(realization, string4)
• institutiondata_type "char"
• institutionstandard_name "institution"
  
• institutionlong_name "Institution
  responsible for the
• 
  forecast system"

10
Ensemble NetCDF File Structure

• CDL Examples (Ensemble parameter example)
• short t(reftime, realization, level, latitude,
  longitude)
• tdata_type "short"
• tunits "K"
• tstandard_name "air_temperature"
• tcoordinates "leadtime institution
  forecast_type"
• tscale_factor 0.00190138198339349
• tadd_offset 253.616560226903
• t_FillValue -32767s

11
Conversion Tool Highlights

• Tool User options
• Pack to 16 bit shorts.
• Select grids based on forecast type
• Perturbed
• Control
• Deterministic
• Control and Deterministic
• All 3 forecast types
• Large file support with NetCDF 3.6.0 or later.

12
Conversion Tool Highlights

• Tool Limitations
• First version
• TIGGE data
• Regular Lat/Lon grids
• Requires Multiple Outside Libraries
• NetCDF
• GRIB API
• Jasper (JPEG 2000)
• Hard Coded Cell Methods
• Slow

13
Conversion Tool Highlights

• Tool Performance
• GRIB-2 Simple Packing to NetCDF 32 BIT
• GRIB-2 size x 2
• GRIB-2 Simple Packing to NetCDF 16 BIT
• Similar size
• GRIB-2 JPEG 2000 to NetCDF 32 BIT
• GRIB-2 size x 8
• GRIB-2 JPEG 2000 to NetCDF 16 BIT
• GRIB-2 size x 4

14
Conversion Tool Highlights
ECMWF sl params, 2x2 Degree Global Grid, simple
packing, converted on 3.4 GHz processor Linux Box
15
Challenges

• Parameter names
• Map available GRIB-2 names to CF names.
• Product Discipline, Parameter Category, Parameter
  Number
• Map dynamically generated names and GRIB-2
  encoding to CF parameter names
• Abstract Definitions
• Not a simple Discipline/Category/Number
  combination.
• Require statistical processing and level
  information.
• Will generally be CF name Cell method
• Get all centers to use this style of parameter
  encoding.
• NCEP only uses the Discipline/Category/Number
  parameter encoding scheme.
• GRIB API doesnt recognize hard coded parameters
  that include statistical processing (e.g. maximum
  temperature).

16
Challenges

• User Tools
• Compatibility with NetCDF ensemble file
  structure.
• Deal with the ensemble dimension.
• Properly ingest, compute valid times.
• Read Coordinate variables to provide ensemble
  member metadata.
• NetCDF 3.6.0 or later required for large file
  support.

17
Future Plans

• Implement on TIGGE portals to provide option for
  NetCDF format data download.
• Generalize converter tool to encompass all types
  of GRIB data.
• Add GRIB API Definitions files for CF parameter
  names, and cell methods.
• Support Additional GRID types
• Explore NetCDF 4
• API with greater functionality and flexibility.
• Add user options.