Observations, PREPBUFR, and the Interface to PREPBUFR 1st GSI User Orientation 46 January 2005 Denni

1
Observations, PREPBUFR, and the Interface to
PREPBUFR1st GSI User Orientation4-6 January
2005Dennis KeyserNOAA/NWS/NCEP/EMC
2
Observational Data Pipeline to NCEP

• GTS and aviation circuit bulletins transferred
  from the NWS Telecommunication Operations Center
  (TOC/NWSTG) to NCEP's Central Operations (NCO)
• Networked to one of two interactive nodes on NCEP
  IBM-CCS (one production, one development) using
  LDM (Local Data Manager) and DBNet (Distributive
  Brokered Networking) software packages
• Data are then encoded into WMO BUFR format

3
Observational Data Pipeline to NCEP (cont.)

• Most of the NESDIS satellite data are processed
  in batch mode as they become available from the
  various NESDIS ftp servers
• Regularly scheduled jobs on the IBM-CCS transfer
  "new" files from these servers and encode the
  data into WMO BUFR format
• NCEP receives Level 3 radial wind data from 154
  NEXRAD radar stations via the radar multicast
  (NIDS format)
• At eight minutes past each hour, a script runs to
  gather up the most recent radar data into eight
  pairs of files containing the raw data and a
  station list
• At thirty minutes past each hour these raw data
  are superobed and encoded into WMO BUFR format

4
Observational Data Pipeline to NCEP (cont.)

• Mesonet surface data are obtained directly from
  FSL
• FSL pushes data to NCO server
• NCO converts data from netCDF to WMO BUFR format

5
BUFR Database on NCEP IBM-CCS

• All of the encoded BUFR data are appended to the
  appropriate files (tanks) in the database
• The files are organized by the WMO BUFR type and
  local subtype and most contain information in 24
  hour blocks (based on report time)
• Observational files remain on-line for up to 10
  days before migration to offline cartridges
• While online, there is open access to them for
  accumulating late arriving observations, for
  operational and checkout dumping, and for
  research and study

6
Network Data Dumps

• The various NCEP production suite networks access
  the observational database at set times each day
  (i.e., the data cutoff time) and perform a
  time-windowed and, in the case of regional
  networks, geographically-filtered dump of
  requested observations
• Observations of a similar type e.g.,
  satellite-derived winds (satwnd), surface land
  reports (adpsfc) are dumped into individual
  BUFR files which maintain the original structure
  of reports
• some interactive quality control is applied
• Reports on NCEP reject list are flagged
• NESDIS/National Oceanographic Data Center (NODC)
  q.c. is applied to marine data
• duplicate reports are removed
• upper-air report "parts" are merged.

7
Network Data Dumps (cont.)

• Some of the BUFR data dump files are re-processed
  into new BUFR files such that they can be used
  properly by subsequent PREPBUFR processing or
  analysis programs
• SSM/I wind speed, total precipitable water and
  rainfall products are superobed onto a one degree
  lat/lon grid (in most networks)
• quikSCAT scatterometer data are q.c.d and then
  superobed onto a one-half degree lat/lon grid
• TRMM TMI rainfall product is q.c.d and then
  superobed onto a one-degree lat/lon grid
• Additional function of dump step is to q.c.
  tropical storm position records (used later in
  PREPBUFR processing)

8
Network Data Dumps (cont.)

• Current data dump (cutoff) times
• ETA (NAM) T111 (00, 12Z), T050 (06, 18Z)
• EDAS (NDAS) 2240/1040 UTC (t-12, t-09, t-06),
  2300/1100 UTC (t-03) for 00/12Z 0520/1720 UTC
  (t-12, t-09, t-06, t-03) for 06/18Z
• AVN (GFS) T246 (00, 06, 12, 18Z)
• FNL (GDAS) T605 (00Z), T805 (12Z), T550
  (06, 18Z)
• RUC (upper-air) T058 (00, 12Z), T026 (all
  other hours)
• RUC (surface) T005 (early, all hours), T022
  (catch-up, all hours)

9
PREPBUFR Processing

• The final step in preparing most of the
  observational data for assimilation by the
  analysis
• Executes a series of programs designed to
  assemble observations dumped from the BUFR
  database
• Encodes information about the observational error
  for each data type as well the background (first
  guess) interpolated to each data location
• Performs both rudimentary multi-platform quality
  control and more complex platform-specific
  quality control
• Stores the output in a monolithic BUFR file,
  known as PREPBUFR

10
PREPBUFR Processing (cont.)

• Tropical cyclone relocation
• A series of programs that relocates one or more
  tropical cyclone or hurricane vortices in the
  global sigma first guess file which is read by
  the analysis

11
PREPBUFR Processing (cont.)

• PREPDATA program
• Reads in and consolidates observations dumped
  from individual BUFR databases
• Performs rudimentary checks on the data, and
  organizes upper-air data by decreasing pressure
• Adds forecast background (first guess)
  interpolated to each observation location
• Adds observational error (read in from a look-up
  table) to each observation
• Converts dry bulb temperature to virtual and
  dewpoint temperature to specific humidity for
  surface data
• Generates an initial pre-q.c. monolithic
  PREPBUFR file

12
PREPBUFR Processing (cont.)

• SYNDATA program
• Reads in quality controlled tropical storm
  position records file valid at the current time
  and uses them, along with other observations in
  the PREPBUFR file, to generate synthetic (bogus)
  wind mandatory level profile reports (throughout
  the depth of the storm) in the vicinity of the
  storm(s) to better define tropical systems for
  the analysis
• Runs only if previous Tropical Cyclone Relocation
  processing was unable to locate a vortex center
  (i.e., cases where the vortex is weak)

13
PREPBUFR Processing (cont.)

• CQCBUFR program
• Performs complex quality control on rawinsonde
  height and temperature data to identify or
  correct erroneous observations that arise from
  location, transcription or communications errors
• Attempts to correct commonly occurring types of
  errors
• Data that cannot be corrected are flagged and
  will not be considered by the analysis
• The checks used are hydrostatic, increment,
  horizontal statistical, vertical statistical,
  baseline and lapse rate and are based upon
  differences from the first guess

14
PREPBUFR Processing (cont.)

• CQCBUFR program (cont.)
• Also applies intersonde (radiation) corrections
  to the quality controlled rawinsonde height and
  temperature data
• The degree of correction is a function of the
  rawinsonde instrument type, the sun angle and the
  vertical pressure level.
• Converts rawinsonde and dropwinsonde dry bulb
  temperature to virtual and rawinsonde and
  dropwinsonde dewpoint temperature to specific
  humidity

15
PREPBUFR Processing (cont.)

• PROFCQC and CQCVAD programs
• Perform complex quality control on wind profiler
  and on Vertical Azimuth Display (VAD) wind data
  from WSR-88D radars, respectively in order to
  identify erroneous data and remove it from
  consideration by the analysis
• The checks used are increment, vertical
  statistical, temporal statistical, and combined
  vertical-temporal and are based upon differences
  from the first guess
• CQCVAD includes a bird migration algorithm

16
PREPBUFR Processing (cont.)

• PREPACQC program
• Performs quality control on conventional AIREP,
  PIREP and AMDAR (Aircraft Report, Pilot Report,
  Aircraft Meteorological Data Relay) aircraft wind
  and temperature data
• The flight tracks are checked, with bad reports
  flagged and duplicate reports removed
• In addition, AIREP and PIREP reports are quality
  controlled in two ways
• isolated reports are compared to the first guess
  with outliers flagged
• groups of reports in close geographical proximity
  are inter-compared using both a vertical wind
  shear check and a temperature lapse check

17
PREPBUFR Processing (cont.)

• ACARSQC
• Performs quality control on MDCRS ACARS aircraft
  wind and temperature data
• Currently only performs simple data bounds checks
• PREPACQC and ACARSQC will soon be replaced with a
  new q.c. module used at FNMOC
• Includes track checking on MDCRS ACARS reports as
  well as all other aircraft types

18
PREPBUFR Processing (cont.)

• Structure of the PREPBUFR file
• Reports are grouped into unique BUFR Message
  Types
• ADPUPA upper-air (RAOB, PIBAL, recco, drops)
  reports
• AIRCAR (MDCRS ACARS aircraft reports)
• AIRCFT (AIREP, PIREP and AMDAR aircraft reports)
• SATWND (satellite-derived wind reports)
• PROFLR (wind profiler reports)
• VADWND VAD (NEXRAD) wind reports
• ADPSFC surface land (synoptic, METAR) reports
• SFCSHP surface marine (ship, buoy, C-MAN
  platform) reports
• SPSSMI SSM/I re-prococessed wind speed, tpw,
  rainfall)
• SYNDAT (synthetic tropical cyclone bogus reports)
• GOESND (NESDIS GOES sounding, retrieval, radiance
  data)
• QKSWND quikSCAT re-processed wind speed

19
PREPBUFR Processing (cont.)

• Structure of the PREPBUFR file (cont)
• Future BUFR Message types
• MSONET (Mesonet surface reports)
• RASSDA (RASS virtual temperature reports)
• GPSIPW (GPS-Integrated Precipitable Water
  Reports)
• Individual reports in a BUFR message are further
  defined by a PREPBUFR report type
• Reports are split into mass and wind pieces
• All mass reports contain report types in the
  range 100-199
• All wind reports contain report types in the
  range 200-299
• Observation errors are defined according to
  report type
• The assimilation uses the PREPBUFR report type to
  identify the reports in the PREPBUFR file

20
PREPBUFR Processing (cont.)

• Structure of the PREPBUFR file (cont)
• Most of the observation types in the PREPBUFR
  file are associated with quality markers
• These are used by the analysis to place a weight
  on the data
• Each PREPBUFR processing step which changes a
  datum (either the observation itself, or its
  quality marker) records the change as an "event
  with a program code and a reason code
• Each time an event is stored, the previous events
  for the datum are "pushed down" in the stack,
  thus the PREPBUFR file contains a complete
  history of changes to the data throughout all of
  the PREPBUFR processing
• The most recent changes are always at the top of
  the stack and are thus read first by any
  subsequent data decoder routine

21
PREPBUFR Processing (cont.)

• Operational Report Types in PREPBUFR file
• 120/220 ADPUPA - rawinsonde virtual temperature,
  specific humidity, station pressure / wind
• 130/230 AIRCFT - AIREP and PIREP aircraft
  sensible temperature / wind
• 131/231 AIRCFT - AMDAR AIRCRAFT sensible
  temperature / wind
• 132/232 ADPUPA - flight-level reconnaissance and
  profile dropwinsonde virtual temperature,
  specific humidity, station pressure / wind
• 133/233 AIRCAR - MDCARS ACARS aircraft sensible
  temperature / wind
• 150 SPSSMI - SSM/I superobed FNOC rainfall
• 151 GOESND - NESDIS SFOV GOES cloud top pressure
• 152 SPSSMI - SSM/I superobed total precipitable
  water

22
PREPBUFR Processing (cont.)

• Operational Report Types in PREPBUFR file (cont)
• 156 GOESND - NESDIS GOES 4-layer PW over land
  clear
• 174 GOESND - NESDIS GPES brightness temperatures
  over ocean clear
• 180/280 SFCSHP - surface marine (ship, buoy,
  C-MAN) virtual temperature, specific humidity,
  station pressure / wind
• 181/281 ADPSFC - surface land synoptic and METAR
  station pressure / wind
• 182 SFCSHP Dropwinsonde splash level virtual
  temperature, specific humidity, station pressure
• 183 ADPSFC, SFCSHP - surface marine (ship, buoy,
  C-MAN), land synoptic and METAR virtual
  temperature, specific humidity (estimated station
  pressure not used)
• 187/287 ADPSFC - surface METAR virtual
  temperature, specific humidity, station pressure
  (estimated from altimeter) / wind

23
PREPBUFR Processing (cont.)

• Operational Report Types in PREPBUFR file (cont)
• 210 SYNDAT - synthetic tropical cyclone wind
• 221 ADPUPA - PIBAL wind
• 223 PROFLR - wind profiler
• 224 VADWND - NEXRAD VAD wind
• 242/252 SATWND - JMA IR visible cloud drift
  wind below/above 850 mb
• 243/253 SATWND - EUMETSAT IR visible cloud
  drift wind below/above 850 mb
• 245 SATWND - NESDIS IR cloud drift wind
• 246 SATWND - NESDIS imager water vapor wind
  (motion)
• 282 SFCSHP ATLAS buoy wind
• 283 SPSSMI - SSM/I superobed wind speed
• 285 QKSWND quikSCAT wind

24
Operational data that do not pass through
PREPBUFR processing

• BUFR dumps for the following data types are read
  directly by the analysis
• NESDIS 1B satellite brightness temperatures from
  TOVS (HIRS-2, MSU) and ATOVS (HIRS-3, AMSU/A,
  AMSU/B)
• NESDIS solar backscatter ultraviolet radiance
  (SBUV) data (ozone)
• NASA Tropical Rainfall Measuring Mission (TRMM)
  rainfall data
• NEXRAD Level 3 radial winds

25
BUFR

• International WMO standard
• Efficient means for transferring data
• Flexibility for adding new observation elements
• Table driven
• Table A defines the data category associated
  with a particular BUFR message containing report
  data
• Table B classifies and defines data elements,
  or descriptors, according to scale, reference
  value, number of bits and units (includes code
  and flag tables)
• Table C defines data description operators
• Table D defines the list of common sequences

26
BUFR (cont.)

• A special application has been designed at NCEP
  to provide user-friendly access to the BUFR files
  through a series of FORTRAN subroutines in a
  machine independent BUFR library (i.e., BUFRLIB)
• Allows data to be encoded into or decoded from
  BUFR using mnemonics to represent the data
• The mnemonics are associated with BUFR
  descriptors in a special version of the BUFR
  Tables A, B, C and D
• When a BUFR file is first created, the mnemonic
  table is read in from an external location and is
  itself encoded into BUFR messages at the top of
  the output file
• This allows each BUFR file to be self defined
  (i.e., no external tables are needed to decode
  data out of the file)

27
Simple Example to Interface to PREPBUFR

• CHARACTER8 SUBSET, STNID
• REAL8 HDR(5), OBS(7,255)
• EQUIVALENCE (STNID,HDR(1))
• .
• CALL DATELEN(10) ! Y2K change all dates
  expressed as 10-digit (YYYYMMDD)
• ! Open BUFR file in unit LUBFR (1st arg) for
  input (read) (2nd arg) using BUFR table internal
  to LUBFR (3rd arg)
• CALL OPENBF(LUBFR,'IN',LUBFR)
• ! Loop through all of the messages in the
  PREPBUFR file
• ! SUBSET returns message type (e.g., ADPUPA)
  IDATE returns message date YYYYMMDD (same for all
• ! messages)
• DO WHILE (IREADMG(LUBFR,SUBSET,IDATE).EQ.0)
• ! Loop through all reports (subsets) in this BUFR
  message
• DO WHILE (IREADSB(LUBFR).EQ.0)
• ! Read header info out of rpt HDR(x) where x 1-
  id (STNID), 2-rpt type, 3-D-time, 4-lon, 5-lat,
  ILVL returns as 1
• CALL UFBINT(LUBFR,HDR,5,1,ILVL,SID
  TYP DHR XOB YOB')
• ! Read level info out of a report OBS(x,y)
  where on level y, x 1-level category,
  2-pressure, 3-temp obs, 4-temp q.m.,
• ! 5-u-wind obs, 6-v-wind obs, 7-wind q.m.,
  ILVL returns the number of levels in the report
• CALL UFBINT(LUBFR,OBS,7,255,ILVL,CAT
  POB TOB TQM UOB VOB WQM')
• .

28
Web Site

• http//www.emc.ncep.noaa.gov/mmb/data_processing/d
  ata_processing/