ICOADS: Data, Products, and Access

1
ICOADS Data, Products, and Access

• Steven J Worley
• Scott D. Woodruff
• Zaihua Ji
• Richard W. Reynolds

2
Outline

• New reference document
• Status of Release 2.2
• Access

3
New Reference Document

• ICOADS Release 2.1 Data and Products
• Worley et al. ,Int. J. Climatol. 25 823-842
  (2005)
• Special Issue Advances in Marine Climatology
• Copies are available
• Release History
• 1st, Release 1, 1985 coverage 1854-1979
• 6th (last), Release 2.1, 2003 coverage 1784-2002
  
• Data Composition
• 185 million records
• Emphasize record tracking - DCK (Table II), SID
  (Table III)
• Development Combination
• Delayed mode (DM) processing - GTS plus other
  sources
• Real-time (RT) mode processing - exclusively GTS

4
New Reference Document

• How DM and RT Compare?
• Use overlap period 1997 ? 1998

Bars DM Lines RT
DM decreases the number of drifting buoy
observation. MEDS, composites and QA checks.
DM increases records for ships, moored buoys, and
C-Man
5
New Reference Document

• Data Usage Considerations
• Changing observing systems - temporally

decreasing ships since 1989 increasing moored and
drifting buoys
6
New Reference Document

• Data Usage Considerations
• Changing observing systems, spatially

45 of possible 2 boxes have 5 samples/month in
the 70-80s
Decreasing low counts due to reduced ships
observations
Increasing high counts due to high frequency
sampling from buoys
7
New Reference Document

• Data Usage Considerations
• Other changes
• Instrument types and sampling procedures
• Codes for measurements and visual observations
• Instrument package placement on platforms
• Sampling frequency and time of day
• More

8
Status of Release 2.2

• Focus time period, 1998 - 2004
• Reprocessing of 1998-2002 from Release 2.1
• Add on two years, 2003-2004
• Processing Status
• Observations
• Complete and available from NCAR (12 October)
• Further analysis of result to be done
• Dont expect a rerun, but .
• Monthly Summary Statistics (MSG)
• Prepare after the observations are fixed ( 1-2
  months)
• LMRF (old binary format) will be generated for
  Release 2.2
• We need LMRF to compute MSG, will transition
  later
• Some past users may want this also - access by
  special request
• Warning LMRF will be discontinued

9
Status of Release 2.2

• Availability
• First from NCAR
• All products
• Following soon after from ESRL
• The former NOAA/CDC is now part of the NOAA Earth
  System Research Laboratory (ESRL)
• Only Monthly Summary Statistics in netCDF format
• Future at NCDC
• Observations retrospective and near real-time

10
Status of Release 2.2

• Data Composition
• New DM Data, 1998-2004
• TAO/TRITON, PIRATA, JAMSTEC Moored Buoy Research
  Archives
• Including RH DPT
• NDBC Moored Buoy Data
• DPT in DM and frequency not known in RT stream
• MEDS Drifting Buoy QCd Archive
• WMO Pub 47 metadata (1973 - 2004)
• Thanks to Liz Kent and Dave Berry
• Kent, et al, 2005 WMO Pub. No. 47 Metadata and
  an Assessment of Observation Heights in ICOADS.
  Submitted JAOT
• http//www.soc.soton.ac.uk/JRD/MET/eck_pubs.php

11
Status of Release 2.2

• Data Composition
• Carry forward RT Data, 1998-2002
• NCEP GTS with DM sources filtered out
• New RT Data, 2003-2004
• NCEP GTS with filtered

12
Status of Release 2.2

• Challenges and Misc. Notes
• Meshing DM tropical moored buoys
• Sampling frequency, hourly and 10-minute
• 10-minute reduced to hourly
• Clean insert with accurate exclusion of GTS
• Daily resolution and matching WMO numbers
• Removed buoys with erroneous IDSHIP from NCEP
  ship GTS
• Eight month period (10/99 - 5/2000)
• Fill some gaps in MEDS with NCEP buoy GTS
• Made some QA(QC) flag fixes for full period of
  record
• Minor errors transitioning from LMRF to IMMA
  format
• Further checking will be done and documentation
  prepared
• Many details are not told here
• www.cdc.noaa.gov/coads/e-doc/other/ltdupelim_1998gt
• Ask questions offline

13
Status of Release 2.2

• Moored buoy coverage
• NDBC (approximate)
• PMEL/NOAA (TAO and PIRATA)
• JAMSTEC
• Note new array beginning in Indian Ocean
• European buoys not shown

14
Status of Release 2.2

• Resource issue for US ICOADS team
• Current personnel

Person Time Commitment Role Name Organization
.8 Coordinator Scott Woodruff NOAA/ESRL
.8 Programmer Sandy Lubker NOAA/ESRL
.1 Coordinator Steven Worley NSF/NCAR
.3 Programmer Zaihua (Hua) Ji NSF/NCAR
.1 Science Advisor Dick Reynolds NOAA/NCDC
.1 Marine Operations NOAA/NCDC
2.2 Total
Bad News FY05 lost 20 of programmer _at_ESRL Bad News FY05 lost 20 of programmer _at_ESRL Bad News FY05 lost 20 of programmer _at_ESRL Bad News FY05 lost 20 of programmer _at_ESRL
Good News Received international contributions (e.g. Kent et al, metadata) NCDC/CDMP (Elms) supporting data and metadata rescue Good News Received international contributions (e.g. Kent et al, metadata) NCDC/CDMP (Elms) supporting data and metadata rescue Good News Received international contributions (e.g. Kent et al, metadata) NCDC/CDMP (Elms) supporting data and metadata rescue Good News Received international contributions (e.g. Kent et al, metadata) NCDC/CDMP (Elms) supporting data and metadata rescue
15
Access

• Links to all data and information
• www.cdc.noaa.gov/coads (under Data and
  Metadata)
• See Table VI (Worley et al. 2005)

Product Format Partner Subset Comments
Observations
Release 2.2 ASCII NCAR t,s,v Binary (LMRF) by request
Release 2.1 ASCII NCDC t,s Currently not available
Auxiliary ASCII, other NCAR none GUI (IMMA), and datasets
NRT ASCII ESRL none NCEP GTS, min. variables, update cycle end of month 5 days
Monthly Statistics
Release 2.1 binary, ASCII NCAR t,s,v GUI and bulk download
Release 2.1 netCDF, ASCII, other ESRL t,s,v GUI or LAS
NRT netCDF, ASCII, other ESRL t,s,v GUI or LAS, min. variables
t,s,v subsetting by time, space, and variable
16
Access

• User profiles from NCAR for the past 14 Months
• Who is using ICOADS?

17
Access

• User profiles from NCAR for the past 14 months
• What quantities of ICOADS are requested?

18
Access

• User profiles from NCAR for the past 14 months
• What time domains of ICOADS are requested?

19
Access

• User profiles from NCAR for the past 14 months
• What space domains of ICOADS are requested?

20
Access

• Review IMMA structure

Mandatory Record Segment
IMMA core, basic data, 48 fields
Optional Record Attachments
ICOADS, historical ID (e.g. DCK, SID), QA(QC)
flags, etc
IMMT-2/FM 13, non-core fields from IMMT and FM-13

In use
Ship Metadata, WMO Pub. 47 ship metadata
Model QC, e.g. UK model comparison with VOSCLIM
To be activated
Historical (proposed, primarily for pre-1854
data)
Others (e.g. moored/drifting buoys?)
Recommended Attachment
Supplemental Data original data in original
format
21
Access

• New interface to IMMA observations _at_ NCAR
• Put online about two weeks ago
• Verify and improve linkages throughout ICOADS
  portal
• Builds on previous successes
• Provide ASCII output
• Previously from LMRF (binary) now IMMA
• All data are available for bulk file download
  (FTP, Web)
• Users select time and space domains
• User select how QA(QC) flags are applied
• Added services
• Access to all attachment data
• Auxiliary data sources

22
Access

• Highlights from Interface
• Filtering application

23
Access

• Highlights from Interface
• Core variable selection

24
Access

• Highlights from Interface
• Attachment data e.g. ship metadata

25
Access

• Highlights from Interface
• Auxiliary Data two types
• New, likely unique
• Replacement, destine to eventually replace other
  data

• Note Can switch off ICOADS request
• Just acquire auxiliary data
• If you provide IMMA data we can serve it
• Incorporate in ICOADS later

New
Replacement
26
Access

• New interface - Level of automation
• Request submission detected automatically by
  daemon, every 2 minutes
• Processing begins immediately, with max.
  processes 3
• Processing may take several hours for long
  temporal requests
• IMMA is not efficient for processing speed
• Users are notified by email and download via Web
  or FTP

27
Access

• Sample with ship metadata from WMO Pub 47

• Type of Barometer
• Aneroid (AN)
• Digital Aneroid (DA)
• Length of Vessel (m)
• Depth of SST measurements (m)
• Height of anemometer (m)

28
Access

• Accessories to go with the data
• IMMA documentation
• Sample FORTRAN code created dynamically to match
  data requested
• Release ID
• Variable definitions

29
Access

• Current strategy
• Delayed mode processing
• File based
• Future possibility
• DB with core, attachments, position, date
• Enable near real-time processing
• Processing, O(minutes)
• Push or pull data delivery
• Could be easily updated (monthly) with GTS

30
Conclusions

• New reference document with many details about
  ICOADS (Worley et al. 2005)
• Release 2.2, 1998-2004 (plus other fixes),
  observations are released to the public.
• Monthly Summary Statistics will be computed soon
• Broaden access in different forms from the
  partners soon
• Data services are going well (judged from NCAR),
  especially internationally.
• IMMA permits more comprehensive access to
  attachment and auxiliary data
• Supporting resources in the US are thin.
  International collaboration has been helpful and
  more is welcomed. Project recognition from the
  science community can help sustain support for
  ICOADS

31
Questions? http//www.cdc.noaa.gov/coads/