NARR User Issues

1
NARR User Issues

• Wesley Ebisuzaki
• CPC/NCEP
• Wesley.Ebisuzaki_at_noaa.gov
• httpwwwt.emc.ncep.noaa.gov/mmb/rreanl/

2
Topics

• Files contents, availability, formats
• Tools BUFR, GRIB
• Obtaining the data NCEP, NCAR, NCDC
• Scripting downloads
• What is reality? Is reanalysis truth? An
  insider's viewpoint.

3
GRIB Files

• For grids (horizontal fields) such as 500 mb Z
• Lambert conformal grid 32 km eta grid (221)
• Have grids of lat/lon, display programs
• Model grid not available except at NCEP
• No plans to make available, display programs?
• Analyses are on-line and forecasts ?
• 99 of people will use the GRIB files

4
BUFR

• For all input observations except SST, snow, ice,
  observed precipitation (GRIB)
• Input obs BUFR files have QC marks
• Analyses/forecasts at station locations (MOLTS)
• Much smaller than GRIB files, limited locations
• Distribution (informal)
• Includes restricted data
• Few users (experience from global reanalysis)
• Often other sources of data (in easier formats)

5
Misc Files

• Restart files huge, full precision, NCEP-only
• Statistics
• Fit of obs to NARR, fit of obs to R2
• Small, can be made available on-line
• Observation counts
• Small, should be put on-line
• PNG (monitoring plots)
• Small, should be put on-line

6
Want more files?Don't expect too much.

• NARR has zero funding.

7
Despair?

• Global Reanalysis has been running this way for
  years!
• CPC has been running global reanalysis for its
  own operations. CPC and the outside community
  have shared interest in maintaining a stable
  assimilation system. NCEP has an interest in
  others using its products.
• However, limited resources, need to be selective,
  work with partners

8
Status

• Real-time is running (4 days late)
• Started building 1.5-2 year on-line rotating
  archive of latest analyses at NCEP
• 1979-2003 merged files at NCAR, NCDC
• NCDC has 5 TB merged files on-line, trying to
  support the on-line community by NOMADS
• EMC people are working on other projects

9
Tools BUFR

• Situation improving
• GrADS can read bufr directly

10
Tools GRIB

• GrADS display and computations
• grib2ctl.pl make GrADS control files
• wgrib inventory, decode GRIB files, remove
  records from a file (save space)
• lcgrib regional subset (save space)
• copygb change grid (save space or neurons)
• grbpoint.pl grib -gt point value printout (slow)
• Gempak (?), NCL (?)

11
Example GrADS

• Make control file (analyses)
• grib2ctl.pl -rr cpc_rr_mon.197901 gtmy.ctl
• grib2ctl.pl -rr cpc_rr_mon.y4m2 gtmy.ctl
• Make index file (analyses)
• gribmap -0 -i my.ctl
• Run GrADS
• grads
• open my.ctl

12
Example wgrib

• Use inventories, make files smaller, decode data
  for fortran and C programs
• See contents of a grib file
• wgrib (options) cpc_mon_197901
• To extract records, decode records, see wgrib
  home page.

13
Example lcgrib

• Regional subset of Lambert conformal grid
• MAKE SMALLER FILES
• lcgrib in-grib out-grib west south east north
• West, south, east and north are in degrees
• N and E are ve, S and W are -ve

14
Obtaining Data NCEP

• Rotating archive latest 1.5 2 years of
  analyses
• http//nomad7.ncep.noaa.gov/ncep_data
• Other files TBD
• NCEP not set up to deal with the public requests
• Go through the web
• Go through NCDC, NCAR

15
Obtaining Data NCAR

• Have 1979-2003 merged data
• Sure they will get more of the data
• Plans, availability?

16
Obtaining Data NCDC

• NCDC has the merged data 1979-2003 and want 2004
  ASAP
• NCDC will use NOMADS to distribute the data to
  the on-line community.
• NOMADS is project of many institutions including
  NCDC and NCEP for distributing data over the
  internet.
• NOMADS-NCDC is taking software from NCEP and COLA
  to distribute GRIB data.

17
Data Access at NCDC

• Custom plots
• ftp4u access by selected time, field, region
• http access to raw data (like FTP)
• GDS (GrADS Data Server) OpeNDAP server (formerly
  DODS), on-line access to data

18
Difficulties in on-line distribution

• Downloading a 5 TB data set is difficult
• Need ways to minimize data transfer
• Plots (maps) work but limited utility
• GDS needs to be optimized for large datasets
• ftp4u is a solution

19
FTP4U

• Only download data that you need
• Select levels
• Select variables
• Select geographic regions
• Select times

20
Making FTP4U Work

• Don't request more data than needed
• Make one big request, not many small requests of
  the same file i.e., make one request to get the
  winds, temp and heights rather than 3 requests.
  Do not request long time series.
• Shared resource (1 job at a time)
• Be prepared to script/automate the downloads
• May be impossible to download in a few hours, may
  require days, weeks and multiple jobs/requests

21
Scripting FTP4U Downloads

• FTP4U is normally accessed by a browser
• Anything a browser can do, a script can also do
• Experience at NCEP is that most plot/ftp4u access
  is scripted (automated)
• NCDC's modifications to the NCEP provided
  programs make it more difficult to script the
  download but nothing that a cURL user couldn't
  handle. NCDC wants to change!

22
Scripting Downloads cont'd

• Use cURL, a unix/windows program, to emulate a
  browser
• Each browser page becomes a call to cURL.
• With a little RTFM was able get script working
• sample program will be put on NARR home page

23
Scripting Downloads cont'd

• Downside steep learning curve, may have no local
  experts for help
• Upside can make large data transfers possible,
  can be used in other projects (ex. getting fcst
  data automatically, skewing on-line opinion
  polls)
• Given that we get more data off the web, it's a
  skill that someone in a large group should
  eventually have.

24
Obtaining Data NCDC

• It's a learning situation for NCDC
• Give feedback (complain), it's the only way to
  improve the situation. Lessons learned here will
  help the next time you try getting a different
  dataset from NCDC's nomad server.
• Current issues full disk (new RAID is will be
  installed soon), obtaining time series, scripting
  support

25
What is Reality?

• Some people consider reanalysis to be truth
• NARR is new, bound to be warts, already found
  some.

26
What is Reality?

• Accuracy of reanalysis depends on many factors
• Data going into the analyses
• Types, counts, quality of observations
• Accuracy of previous analysis
• Accuracy of atmospheric/land model
• Spatial and temporal filtering
• NO EASY ANSWER

27
What is Reality? cont'd

• Accuracy
• check to the appropriate data counts and biases
• Need to add our counts to web
• Need to add fit to data to web
• check other reanalyses
• Fit to observations
• Consider sensitivity to modeling assumptions
• Sensitivity experiments (data, resolution,
  model), only available to limited of people

28
What is Reality? cont'd

• Trust but verify
• Ronald Reagan

29
URLs

• copygb ftp//ftpprd.ncep.noaa.gov/pub/cpc/wd51we/
  copygb
• change grid type of grib files
• copygb ftp//ftpprd.ncep.noaa.gov/pub/cpc/wd51we/
  copygb
• change grid type of grib files
• cURL http//curl.haxx.se
• download URLs, handles post data
• GEMPAK http//my.unidata.ucar.edu/content/softwar
  e/gempak/index.html
• graphics, can handle NARR files with some effort
• GrADS http//grads.iges.org/grads/
• graphics, computations

30
URLs

• grbpoint.pl ftp//ftpprd.ncep.noaa.gov/pub/cpc/wd
  51we/wgrib.scripts
• simple program to extract data from lat/lon
• grib2ctl.pl http//www.cpc.ncep.noaa.gov/products
  /wesley/grib2ctl.html
• makes control files for GrADS
• g95 http//www.g95.org/
• f95 compiler for copygb
• HIPHOP http//iacweb.etzh.ch/stall/dominik/hihop/
  
• grib support for idl
• lcgrib http//www.cpc.ncep.noaa.gov/products/wesl
  ey/gribw.html
• regional subset of Lambert conformal grib files

31
URLs

• NARR http//wwwt.emc.ncep.noaa.gov/mmb/rreanl/ind
  ex.html
• home page for NARR
• NARR rotating archive http//nomad7.ncep.noaa.gov
  /ncep_data
• NCAR Command Language (NCL) http//ngwww.ucar.edu
  /
• graphics, computations package that supports grib
• NCDC data http//nomads.ncdc.noaa.gov/narr_datas
  ets
• on-line supplier of NARR data
• wgrib http//www.cpc.ncep.noaa.gov/products/wesle
  y/wgrib.html
• inventory, slice dice grib files