NOAA Operational Model Archive and Distribution System

1
(NOAA Operational Model Archive and Distribution
System)

For Access to Operational Model Data and Value
Added Products Real Time NOMADS
Jordan C. Alpert jordan.alpert_at_noaa.gov Jun Wang
jun.wang_at_naa.gov Glenn
Rutledge glenn.rutledge_at_noaa.gov NOAATECH
10/23//03
where the nations climate and weather services
begin
2
NOMADS is an agreement between agencies who
participate to have common

• Data and observation distribution software,
  format independent and description methodology,
• Documentation and organization framework,
• Forum to plan and organize
• Funding avenue to direct .
• for university, federal agencies and
  organizations, a mechanism to obtain support for
  the dissemination of their data sets.

3

Participants U.S.- NOAA, DOE, NASA, NCAR,
Unidata, COLA, OPeNDAP Project, Several
Universities International- CEOS NERC ESG UK
BADC BOM / Dept. of State via CAP
4
NOMADS and the NCEP Real Time Component
A solution for Recommendation 5 of Fair Weather
Effective Partnership in Weather and Climate
Services

• Wider information dissemination through web pages
  and other avenues.
• Framework to create specialized and value added
  products needs access to weather data.
• Facilitate communication among commercial,
  public, research community and federal agencies.
• NOMADS is a system for participants to interact
  with others using common protocols and formats.
  Existing differences in details are mollified by
  software.

5
RT-NOMADS at NCEP
Consists of

• Distributed Ocean Data Server - OPEN_DAP (DODS)
  based on metadata file descriptors and XML.
• Web based ftp, the NCEP ftp2u client and Great
  Displays (pdisp), a grid analysis and display
  client.
• Model initialization ingredients, The minimum
  set for model rerun/retrospective analysis ready
  observations restart files.
• The Real Time NCEP Operational model run history.

6
RT-NOMADS is a service for science, development
and public access.

• To distribute NCEPs Operational data set to
  researchers and the public.
• To provide server software to NOAA projects who
  need to serve their data through web services.
• NCEP NOMADS servers are the intermediary between
  NCEP operational computers and NCDC NOMADS
  archive servers.

7
Tools for Users on the server

• Pare down large file sizes of high resolution
  initialization and model results.
• (re-) Group different data sets to create needed
  products such as initialization files for model
  development.
• Subset the data sets in parameter space
• Subset the data sets in physical space
• (subset the grid in 4 dimensions)

8
NCEP Real-Time NOMADS Server

• Real time.
• Operational Suite of Model products.
• Forecast information for model comparison.
• Operational initialization and observations.
• Maintenance of codes.
• Utilities for data manipulation and grid
  transformation sub-setting.
• Documentation.
• Serves as transfer point to NCDC for NOMADS
  archive of reruns and retrospectives.

9
The Server Client distributed relationshipA
Common Sense Approach to Data Distribution
Examples of Clients

• ftp2u, Great Displays, and GDS allow sub-setting
  of data by parameter, by 3D region, and by time.
• Allows for results with low band width! Eg.,
  NASA/GSFC takes 1 hour to ftp entire GFS (AVN)
  forecasts or 13 seconds to download their
  desired fields (ftp2u).
• Great Displays (pdisp) A display program
  operating from the same database as ftp2u to
  display any part of the NCEP data base by time,
  region, vertical level and variable.

10
The GrADS-DODS Server (GDS)

• NOMADS participants serve their data sets
  through a client-server relationship, that is,
  the data sets are internet ready and the display
  is done by the client.
• GDS combines both GrADS, a freeware client (from
  COLA) and DODS (OPeN-DAP) server to compress and
  exchange data in many formats with http.
• This means that data appears to the user or
  client application as a local file!
• Like Network file system
  over the web!
• DODS requests are made by many freeware and
  commercial high level language clients.
• Simple http queries to the DODS server can create
  value added products.

11
Data Application Protocol (DAP) using
12
A GrADS client example.
Sdfopen http//nomad29090/dods/reanalysis-2/pgb/h
gt t 1 'set gxout shaded' while (t lt 1068)
'set t 't 'q time' datesubwrd(result,3)
'draw title hgtprs 'date' 't 'd hgtprs'
'/home/wd23ja/grads/lib/cbarn.gs' 'swap' t
t 1 endwhile
13
Using http to extract data from the server

• http//140.90.198.1569090/dods/eta/eta20030527/et
  a_00z.ascii?tmpprs0000130130 290290
• the order of the square bracked values is
• timelevellatlon
• where lat is measured from the south pole (0) to
  NP (180)
• In units represented by the metadata descriptor
  file.
• (Use a wget and the URL in cron or a cgi-bin
  script provides needed values.)

14
MATLAB A Commerical Client

• servernomad29090
• directory/dods/gdas/rotating/ fextascii
• modelgdas datestrYYYYMMDDHH
  varnamehgt
• varlevs1
• Arrc,t1,,t2,,nlev1,,nlev2,,ilat
  1,,ilat2,,ilon1,,ilon2,
• urlstringhttp//,server,directory,model,datestr
  ,.,fext,?,varname,arrc
• url java.net.URL(urlstring) connect to
  web page
• Try
• stream openStream(url)
• ireader java.io.InputStreamReader(stream)
• breader java.io.BufferedReader(ireader)
• Line readLine(breader) disp(line)
  

15
NOMADS Data Sets at NCEP
Metadata descriptor files describe all aspects of
the data sets and are created uniquely from the
headers of GRIB files. Data sets at NCEP

• Real Time Operational model data sets from GFS,
  and ETA from present to one month ago.
• Attention to accumulation variable differences
  and high and low resolution.
• Analysis pgb, restart files.
• Recent (real time) climate runs, ie., RSM.
• Observations (soon).
• Reanalysis-2 (and 1)
• AMIP type runs (Surus finest runs eg., 28 and 64
  levels 1978-2002), sudden warming, .

16
. Data Sets (continued)
Archives at NCDC

• NOAAPORT Operations.
• Continuation of NCEP real time model component.
• NCEP model initialization ingredients, The
  minimum set for model rerun/retrospective
  analysis ready observations restart files.

17
Model Data Reqs - NWP and GCMs
GCM- Global Climate Model Reanal- N. Am. Regional
Reanalysis NWP- Numerical Weather Prediction
5-year rotating NWP archive
Alpert Rutledge 6/03
18
RT NOMADS Server at NCEP

• Two 1TB RAIDZONE servers
• http//nomad2.ncep.noaa.gov
• http//nomad1.ncep.noaa.gov (backup)
• Documentation web page under construction

19
Value Added Products and the National Digital
Forecast Database

• Value added Retailers who make value added
  products can use NOMADS GDS to get the
  meteorological data they need without downloading
  entire files.
• Users (read forecasters) of NDFD can create their
  own products using GDS server accessing only data
  they need.
• GDS reduces the bandwidth needed to create
  products in weather service operations.
• For internet-2 bandwidth, servers at Regional
  Centers can distribute data to WFOs for their
  operations.

20
Demonstration