THREDDS, CDM, OPeNDAP, netCDF and Related Conventions

1
THREDDS, CDM, OPeNDAP, netCDF and Related
Conventions

• John Caron
• Unidata/UCAR
• Sep 2007

2
Contents

• Overview
• THREDDS Data Server
• Unidatas Common Data Model

3
1) Request
0) Client
2) Server
3) Response
4
0) The Client

• What functionality is needed?
• Scientific User
• Raw data
• Drill down to arbitrary detail
• Decision Support
• best effort Visualization
• operational

5
1) The Request

• What functionality is possible?
• Analogous to SQL language for RDBMS
• Implies a Data Model
• OGC vs File access APIs
• NetCDF/OPeNDAP/HDF5 index space
• WXS coordinate space
• Higher semantic level trumps if no significant
  extra cost.
• File APIs become implementation, not interface

6
1) WCS Request

• Functionality
• Subsetting (bounding box, time range, variable)
• Optional reprojection/resample
• Variants KML/XML/SOAPXML/REST
• Optional Functionality 42 flavors
• Bad news for interoperability
• Is there an elephant to dictate a standards?
• Eg IBM chose SQL/Relational model (1984)

7
2) Server

• How do I serve my data?
• Do I need specialized personnel?
• , resource consumption, core competency
• What are the common requests?
• (that I should optimize for)?

8
3) Response

• What comes back?
• Has to be a representation of the answer in the
  Data Model
• WCS allows anything
• Cant write a generic client
• Communities will form around a small number of
  variants
• No elephants in sight

9
3) Response XML vs. binary

• Extensibility vs. Efficiency
• Binary netCDF/GeoTIFF/HDF/etc
• reflect favorite formats of committee members
• Different data models ideally need a formal
  mapping (but there arent any yet)
• Domain experts can make use
• GML closely follows the OGC/ISO data model (WFS
  requires GML)

10
3) Response XML vs. binary

• GML is waaaay too complex
• Ambitious
• OGC/ISO models are complex
• Reality is complex
• XML Schema is a disaster
• Google KML
• visualization format not data storage

11
THREDDS Data Server
HTTP Tomcat Server
catalog.xml
Application
THREDDS Server

• WCS

• OPeNDAP

• HTTPServer

• NetcdfSubset

NetCDF-Java library
configCatalog.xml
Datasets
IDD Data
motherlode.ucar.edu
12
THREDDS Catalogs

• XML over HTTP
• Hierarchical listing of online resources
  (datasets)
• Container for arbitrary search metadata
• Standard set maps to DC, GCMD, ADN
• Unidata/NCAR-CDP
• Metadata can be inherited
• Design goal Make it easy for data providers
• TDS uses extended version for configuration
• Data Access URLS
• Crossing the protocol boundary

13
THREDDS OPeNDAP Server

• OPenDAP is protocol for remote access to CDM
• Current version 2.0 NASA ESE standard
• Working on new 4.0 protocol spec
• Based on Java-OPeNDAP library
• shared development by Unidata/opendap.org
• Any CDM dataset can be served
• Server4 (Hyrax)
• latest version of opendap.org C library
• THREDDS Catalogs replace dods_dir

14
THREDDS WCS service

• CDM files that have Grid coordinate system
• evenly spaced x,y
• Allow to subset the dataset by
• Lat/lon or projection bounding box
• time and vertical coordinate range
• list of Variables
• Return formats
• GeoTIFF floating point, grayscale
• NetCDF/CF-1.0
• No reprojections, resamplings
• Uses WCS 1.0, work on WCS 1.1 in progress

15
NetCDF Subset Service

• Experiment with REST style web service
• Allow to subset the dataset by
• Lat/lon bounding box
• time and vertical coordinate range
• list of Variables
• NetCDF/CF, XML, CSV (spreadsheet)
• Gridded Data
• Output is a CF-1.0 netCDF file
• Variation of WCS (simplified request protocol)
• Grid as Point Datasets (experimental)
• Extract vertical profile, time series from one
  point in model data
• Station Data metars (7 day rolling archive)

16
(No Transcript)
17
Common Data Model
HTTP Tomcat Server
catalog.xml
Application
THREDDS Server

• WCS

• OPeNDAP

Then a miracle happens

• HTTPServer

• NetcdfSubset

NetCDF-Java library
hostname.edu
Datasets
IDD Data
18
Application
Scientific Datatypes
Datatype Adapter
NetCDF-Java architecture
NetcdfDataset
CoordSystem Builder
NetcdfFile
I/O service provider
OPeNDAP
NetCDF-3
NIDS
GRIB
NetCDF-4
NcML
HDF5
GINI
Nexrad
DMSP

19
Common Data Model File Formats

• General NetCDF, HDF5, OPeNDAP
• Gridded GRIB-1, GRIB-2
• Radar NEXRAD level II and level III, DORADE,
  Chinese NEXRAD
• Point BUFR
• Satellite DMSP, GINI
• In Progress NetCDF4, McIdas AREA, NPOESS, NOAA
  CLASS legacy files, Barrowdale DataBlade, others

20
Common Data Model Layers
Coordinate Systems
Data Access
21
Common Data Model (Data Access Layer)
22
Coordinate Systems UML
23
NetCDF-4 file format

• NetCDF-4 C library
• 4.0 Beta implements CDM access layer
• Persistence format for complete CDM
• 4.1 adding Coordinate Systems
• Optional layer, focus on CF-1 (libcf)
• 4.? merge OPeNDAP access
• NetCDF-Java library will read, maybe write

24
TDS / NcML aggregation

• ltdataset name"WEST-CONUS_4km Aggregation"
  urlPath"satellite/3.9/WEST-CONUS_4km"gt
• ltnetcdf gt
• ltaggregation dimName"time" type"joinNew"gt
• ltscan location"/data/ldm/pub/satellite/3.9/
  WEST-CONUS_4km/" suffix".gini" /gt
• lt/aggregationgt
• lt/netcdfgt
• lt/datasetgt

25
Forecast Model Run Collection (FMRC)
26
Scientific DataTypes

• Based on datasets Unidata is familiar with
• APIs are evolving
• How are data points connected?
• Intended to scale to large, multifile collections
• Intended to support specialized queries
• Space, Time
• Intend to create standard NetCDF file encoding
  conventions

27
Scientific DataTypes

• Grids
• Structured
• Swath
• Unstructured
• Point Observation
• Unconnected
• Station / Time Series
• Trajectory
• Profile
• Radial

28
Climate and Forecast (CF) Conventions

• Conventions for encoding coordinate systems,
  other semantics in netCDF
• Working for 10 years
• Version 1.0 in 2003
• Good for gridded data
• Current working goups
• Point/Station/Trajectory/Profile observations
• CRS (map to OGC)
• Governance in place
• Volunteer motivated, practical, real

29
Summary Unidatas directions

• Client both Scientific User and Decision Support
• Request in coordinate space
• WCS is fine, not a big architectural decision
• Server TDS
• Files in native format, augmented by indexing/DB
• Response netCDF/CF and GeoTIFF/KML or WMS/JPEG