OPeNDAP Tutorial

1
OPeNDAP Tutorial Presented at theStennis Space
Center29-31 September 2004 by Peter Cornillon
and James Gallagher OPeNDAP 165 Dean Knauss
Drive, Narragansett, RI and Jonathan
Callahan Joint Institute for the Study of the
Atmosphere and Ocean (JISAO) University of
Washington, Seattle, WA 98195-4235
2
Outline
3
OPeNDAP Overview Cornillon - Tuesday AM
Active participation
4
Download the OPeNDAP Data Connector

• In your web browser, go to http//opendap.org/
• Click on Download
• Click on Generic Unix following Download 2.57
  under Clients/OPeNDAP Data Connector
• Save the tarball in your home directory
• Unzip and untar the tarball
• promptgt tar xzf ODC-2.57-unix.tar.gz
• cd into the ODC directory that it creates
• promptgt cd ODC-2.57
• Start the ODC
• promptgt sh startup

5
Demo

• Matlab
• Graphical User Interface (GUI)
• Command Line Client scripting
• The OPeNDAP Data Connector (ODC)
• Finding data
• Accessing data
• Plotting data

6
Philosophical underpinnings and history of
OPeNDAP The Open source Project for a Network
Data Access Protocol
7
Data systems generally involve some combination
of the following elements
Archive
Access/ Delivery
8
In the Past
Historically these elements have been developed
and managed as monolithic systems by the groups
assembling the elements.
9

• The trend is away from centrally designed,
  implemented and maintained systems

• toward
• The integration of independently designed,
  implemented and maintained system elements.

10
(No Transcript)
11
The OPeNDAP data access protocol has been
conceived as an access/delivery element in this
environment of distributed data system elements.
12
(No Transcript)
13
The DAP

• The OPeNDAP philosophy focuses on the development
  of a highly modular system based on the OPeNDAP
  data access protocol (DAP).

• The DAP allows system elements to talk to one
  another its the glue between system elements.

14
Some History
15
Distributed Oceanographic Data System (DODS)

• Conceived in 1992 at a workshop held at URI.

• Objectives were
• to facilitate access to PI held data as well as
  data held in national archives and
• to allow the data user to analyze data using the
  application package with which he or she is the
  most familiar.

• Basic system designed and implemented in
  1993-1994 by Gallagher and Flierl.

16
Distributed Oceanographic Data System

• DODS consisted of two fundamental parts
• a discipline independent core infrastructure for
  moving data on the net,
• a discipline specific portion related to data
  population, location, specialized clients, etc.

17
DODS ? OPeNDAP NVODS

• To isolate the discipline independent part of the
  system from the discipline specific part, two
  entities have been formed
• The Open Source Project for a Network Data Access
  Protocol (OPeNDAP)
• The National Virtual Ocean Data System (NVODS)

18
OPeNDAP

• OPeNDAP is a 501 c(3) non-profit corporation

• Formed to maintain, evolve and promote the
  discipline neutral Data Access Protocol that was
  the DODS core infrastructure

19
Considerations with regard to the development
OPeNDAP

• Many data providers

20

• The DAP has been designed to be as general as
  possible without being constrained to a
  particular discipline or world view.

• The DAP is a discipline neutral data access
  protocol it can be used in astronomy, medicine,
  earth science,

21
OPeNDAP System Elements
The OPeNDAP data access protocol is used by a
variety of system elements.

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

22
Servers

• Servers receive requests and provide responses
  via the DAP.

• Servers convert the data from the form in which
  they are stored to the OPeNDAP data model.

• Servers provide for subsetting of the data.

23
OPeNDAP Servers
24
OPeNDAP/NVODS Server Sites
25
Servers

• Servers may also provide other services
• Directory traversal.

• Browser-based form to build URL.

• ASCII version of data.

• Metadata associated with the data.

26

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

27
Clients

• Clients make requests and receive responses via
  the DAP.

• Clients convert data from the OPeNDAP data model
  to the form required in the client application.

28
OPeNDAP Clients
29

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

30
Aggregation Servers

• Aggregation Servers receive requests and make
  responses via the DAP
• and they may
• make requests and receive responses via the
  DAP.

• An aggregation Server may be a client-server
  pair, a gateway, or simply a server.

31
Aggregation Servers

• Aggregation Servers provide access to multi-file
  data sets as though they were single data objects.

• An Aggregation Server has been built by UCAR for
  grids and arrays.

• The JGOFS server is effectively an Aggregation
  Server for sequences.

32
The Aggregation Server An Example
netCDF Data Set
DSP Data Set
33

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

34
Processing Servers

• Processing servers receive requests and make
  responses via the DAP
• and they may
• make requests and receive responses via the
  DAP.

• A processing server may be a client-server pair
  or simply a server.

35
Processing Servers

• Processing servers perform operations on the data
  in addition to transformation to the OPeNDAP data
  model and subsetting, e.g.
• Average
• Sum

36
Processing Servers

• There are currently two examples of processing
  servers
• GrADS Data Server (GDS)
• Ferret Data Server (FDS)

Callahan Thursday AM
37
The Processing Server An Example
38

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

39
Ancillary Information Services (AIS)

• The OPeNDAP data access protocol does not mandate
  semantic metadata content. This was done
• So as not to constrain use of the protocol to a
  specific discipline,

• To provide for maximum flexibility in the
  development of the protocol, and

• To make it as easy as possible to serve data.

40
Ancillary Information Services (AIS)

• This leads to a lack of semantic consistency from
  data set to data set.

• The AIS is a mechanism to provide complete,
  consistent metadata descriptions for data sets
  after the fact.

41
Ancillary Information Services (AIS)

• The AIS allows for the integration of metadata
  into the OPeNDAP data stream from a source other
  than the client or the data server i.e.,
  metadata developed by someone other than the data
  provider or the user.

• At present the AIS is specifically for the
  addition of attributes.

42
The Ancillary Information Services
43

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

44
Browser Interfaces

• Browser interfaces provide rendered images of the
  data. Those of interest here are ones that rely
  on OPeNDAP to access the data rendered.

• They use a two-tier architecture
• HTML/GIF,JPEG, for communication with the user.
• OPeNDAP for access to the data.

45
Browser Interface
46
Browser Interfaces

• Browser interfaces fall into two classes
• GIS based Require GIS metadata
• ArcIMS
• MapServer Uses AIS for metadata
• EASy

• Non-GIS based Use local metadata
• Live Access Server

Callahan Thursday AM
47

• Servers
• Clients
• Aggregation Servers
• Processing Servers
• Ancillary Information Services
• Browser Interfaces
• Data System Integrators (ODC)

48
Data System Integrators

• In an environment consisting of highly
  distributed system elements, the notion of a data
  system is amorphous at best.

• The data system integrator is an element which
  assembles a suite of data system elements that
  together provide seamless access to the data from
  discovery to use.

49

• The data system integrator defines the data
  system.

• The data system integrator generally speaks
  several different protocols.

50
The data system integrator brings order to a
disordered array of system elements.
51
The ODC
52
The ODC
- a Data System Integrator
53
(No Transcript)
54
The OPeNDAP Data Model

• The data access protocol consists of a data model
  and its implementation.
• Data served via OPeNDAP are reformatted into a
  form that is consistent with the OPeNDAP data
  model.

55
The OPeNDAP Data Model

• The OPeNDAP data model is made of a small number
  of low-level atomic'' data types (based on C
  data types) and a few higher order constructor
  types.
• Atomic data types supported
• Byte is a single byte of data.
• Int32 is a 32 bit two's complement integer
• UInt32 is a 32 bit unsigned integer.
• Float64 is the IEEE 64 bit floating point data
  type.
• String is a sequence of bytes terminated by a
  null character.
• Url is a string containing an OPeNDAP URL.

56
The OPeNDAP Data Model

• Constructor types supported
• Array is a one dimensional indexed data structure
  as defined by ANSI C.
• Grid is an association of an N dimensional array
  with N named vectors.
• Structure is a class that may contain several
  variables of different classes. The structure
  type can also be used to group a set of unrelated
  variables together into a single dataset.
• Sequence is an ordered set of variables each of
  which may have several values.

57

• The DAP also includes
• Operators and
• An external data representation
• A procedure for retrieving data and ancillary
  data from remote platforms and
• An API consisting of OPeNDAP classes and data
  access calls designed to implement the protocol.
• The OPeNDAP core library has been implemented
  both in C and in Java.

58
The Data Request

• Data are requested with a URL.
• http//www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Re
  ynolds_sst
• Protocol Machine name OPeNDAP server
  Directory File name

?sst10100900180 Constraint

• The user can impose a constraint on the data that
  s/he would like to acquire from a data set by
  appending a constraint expression to the end of
  the URL.

59
(No Transcript)
60
The Ultimate Objective of a Data System
For example A user might want all ocean
temperature values (with associated times and
locations) that lie between 90 and 110 m and have
uncertainties less than 1ºC.
61
Interoperability

• To achieve this objective, system elements must
  interoperate i.e., the system must
• Be capable of finding all data of interest.

62
Metadata

• These interoperability requirements require in
  turn descriptions of the data i.e., metadata.

The degree of system interoperability is
determined by the associated metadata.
63
Syntactic and Semantic Metadata
64
Syntactic and Semantic Metadata
65
Syntactic and Semantic Metadata
Temperature (ºC) 0 4 8 12 16 18

• 20 30 40
• Time (Hours)

66
Interoperability and OPeNDAP
67
(No Transcript)
68
OPeNDAP Data Types

• OPeNDAP provides three basic data objects
• data descriptor structure (.dds) - syntactic
  metadata - rigid

• data attribute structure (.das) - semantic
  metadata - flexible

• data (.dods) - the actual data in a binary
  structure

69
Data Descriptor Structure

• The DDS is the syntactic metadata for a data set.
  
• The DDS is rigidly constrained.
• This metadata is held in containers within the
  DDS that associate the metadata with the
  variables in the data set.
• The DDS is obtained by the server from the data
  set.

70
Getting the dds

• The DDS may be requested from the browser by
  appending .dds to the data set name.
• http//www.cdc.noaa.gov/cgi-bin/nph-nc/da
  tasets/Reynolds_sst
• Protocol Machine name OPeNDAP server
  Directory File name

.dds
71
Data Attribute Structure

• The DAS contains the semantic metadata for a data
  set.
• There is no constraint on the metadata that is
  placed here.
• The DAS could be empty or could hold an FGDC
  compliant description of the data set.
• At a minimum we would like to see semantic use
  translational metadata here.

72
Data Attribute Structure (cont)

• These metadata are held in containers in the DAS
  that associate the metadata with the variables in
  the data set.
• The DAS may come from metadata bound to the data
  set (e.g., NetCDF), an ancillary DAS file or an
  Ancillary Information Services (AIS) site.
• The DAS for a data set may be requested from the
  browser by appending .das to the data set name.

73
Getting the das

• The DDS may be requested from the browser by
  appending .das to the data set name.
• http//www.cdc.noaa.gov/cgi-bin/nph-nc/da
  tasets/Reynolds_sst
• Protocol Machine name OPeNDAP server
  Directory File name

.das
74
OPeNDAP Server Services
Services obtained by appending the given string
to the URL immediately following the file name

• .ascii an ascii representation of the data.

• .info a more readable version of the dds and
  das combined.

• .html form a web based form that will help to
  build an OPeNDAP URL.

• dods-dir a way of obtaining a listing of
  OPeNDAP-accessible files at a site no file name
  or constraint URL ends in a slash, /.

75
BREAK
76
OPeNDAP Interfaces
We define three classes of interfaces based on
the difficulty involved in building a URL. These
are, in order of increasing ease-of-use

• Command line - One types a fully constrained URL

• General purpose URL builder - One fills in a form
  and a URL is built. Does not require any semantic
  metadata

• General purpose URL builder - One fills in a form
  and a URL is built. Does not require any semantic
  metadata

• Graphical User Interface - Makes use of semantic
  metadata to generate URLs based on geophysical
  variable selection made by the user.

77
Peeking at Data in your Browser
78
Peeking at Data in Your Browser
We will look at the characteristics of the
Reynolds optimally interpolated SST data set
using our web browser. This data set is used as
an example in the Quick Start Guide.

• Go to opendap.org in your browser
• Click on Data Sources
• Click on NOAA/Climate Diagnostic Center (CDC)
• Mouse over the buttons to the left of
  Reynolds_SST
• Note that the title of the button appears if you
  dwell on a button. Were looking for Directory.
  Click on it.
• A new page comes up with a listing of the
  contents of a directory

79
Peeking at Data in Your Browser

• Note that this listing is entitled DODS Index of
  The links on this list are passed through the
  server when selected.
• Click on the sst.mean.nc link.
• A form comes up.
• The URL in the Data URL window is the same URL
  that is used in the Quick Start Guide.
• Go to the quick start guide at
• http//opendap.org/user/quick-html/quick_1.html
• and follow the steps Section 1 and 1.1.

80
Looking at These Data in the ODC
81
Looking at These Data in the ODC
Now, we will examine the Reynolds optimally
interpolated SST data set in the ODC. But first
some background

• The ODC is designed to move the user from
  discovery to display. The three relevant windows
  (steps) are
• Search used for the data discovery process.
• Retrieve used to define a subset of the data.
• View used to plot, or to provide listing of,
  the data.

82
(No Transcript)
83
Looking at These Data in the ODC
Lets do it.

• Start the ODC.
• On startup you will be in the Search window under
  the Dataset List tab. This is the same list as
  we used to find the Reynolds data set in the
  first place. It is traversed similarly here.
• Find the Reynolds data set on this list.
• Double click on the data set name. You are
  automatically passed to the Retrieve window.

84
Looking at These Data in the ODC

• The Retrieve window has two main panels
• Datasets
• Additional Criteria

• Under Datasets there are three panels
• One at the top with all selected data set names
• One under this panel and to the left with the URL
  for the selected data set in it, and
• One under and to the right with, in this case,
  the files in the associated directory.
• You can use these panels to traverse a directory
  structure. Well do this later.

85
(No Transcript)
86
Looking at These Data in the ODC

• Double click on sst.mnmean.nc This reads the
  .dds and .das and lists the variables in the
  data set in the Additional Criteria panel.
• Check the sst box. This provides a means to
  define a subset of this variable. This is all
  determined from the dds
• Click on the Details box at the bottom of this
  frame to display the das information.

87
Looking at These Data in the ODC

• Now lets define a subset
• Choose a 10 step range in time by entering a
  number between 0 and 243 in the left hand box and
  this number plus 10 in the right hand box.
• In the box near the bottom labeled step, change
  the 1 to 2.
• In the step box next to time, change the 2 back
  to 1.
• Click Output to. The default is to go to the plot
  panel of the View window.

88
Looking at These Data in the ODC

• In the plot panel of the View window start by
  taking all the defaults Click Plot to.
• Now change the parameters
• Ask for a coast line
• Click the Options tab of the lower panel.
• Click on the Show Coastline entry in the pane on
  the left hand side of the new frame
• Click the Yes button under Show Coastline.

89
Looking at These Data in the ODC

• Changing the parameters (continued)
• Next correct the temperature scale
• Click on the Retrieve tab.
• Click on the details box to get the scale_factor
  and add_offset parameters.
• Click on the View tab.
• In the Linear Bias box of the Legend/Colorbar
  frame, click the no adjustment box.
• Enter the Slope and Intercept obtained from the
  Retrieve window.

90
Looking at These Data in the ODC

• Changing the parameters (continued)
• Finally change what to plot where
• Click on the Variables tab.
• In the constraint box for time replace 1 with
  1-11.
• Change Preview Pane to Full Screen in the box
  immediately to the right of the Plot to button.
• Now plot it Click the Plot to button.
• Hit any key to get back to the plot control
  panel.

91
Extra CreditFinding a New Data Set in Google
and Looking at It in ODC
92
Finding a New Data Set in Google
Recall that DODS Index of appeared at the top of
the listing of data sets that we obtained when we
clicked on Reynolds SST in the browser. Lets see
what happens when we search on this

• Go to google.com
• Enter DODS Index of in the search window.
• Many of the hits are already on the data set
  list, but
• DODS Index of /data/oceanwatch/nrt/goes
• is not. Click on it.

93
Finding a New Data Set in Google

• The result is a DODS directory listing. Click
  on any of the file names ending in .nc.
• This brings up a form in the browser.
• Capture the URL for this data set from the Data
  URL box as you did when peeking at data in your
  browser.

94
Enter the New Data Set in the ODC
We now have the data set name of interest. Lets
enter it into the ODC.

• Click on the Retrieve tab in the ODC.
• Put the captured URL in the location box at the
  top of the panel This is an alternate way of
  entering URLs in the ODC.
• Strip off the file name at the end.
• Click the Add to list button.
• Enter any name in response to the question.

95
Enter the New Data Set in the ODC

• This will bring up the directory structure for
  the new site.
• Select any of the data sets.
• Subsample BE VERY CAREFUL HERE THESE ARE VERY
  LARGE FILES!
• Pass to the plotter and plot.

96
Exploring the PFEL Site
We looked at one directory at this site. Do they
have more data?

• Back to the Retrieve window.
• Remove the file name, constraint and all of the
  subdirectories from the URL in the Location box
  back to data.
• Add to list
• Give it some name.
• Youre now pretty close to the top of the
  archive.
• Drill down and learn about some data set.

97
Exploring the PFEL Site Conclusion
In addition to learning how to look at data in
your browser and in the ODC, two things have
become apparent in this exercise

• There are sites with data of interest that are
  difficult to find.
• It is hard to find data within a site.

We are working with UCAR to address this with
THREDDS catalogs.
98
OPeNDAP Documentation
99
OPeNDAP User Documentation

• DODS Quick Start Guide - Getting started with
  OPeNDAP.
• DODS Server Installation Guide - Installing an
  OPeNDAP server.
• DODS User's Guide A description of OPeNDAP, and
  the information needed to use an OPeNDAP client,
  as well as to create OPeNDAP clients, convert
  existing applications into OPeNDAP clients, and
  set up OPeNDAP servers.
• DODS C Programmer's Guide and DODS C
  Programming Reference - Provides information
  about the core software, programming tools, and
  DAP classes.

100
OPeNDAP User Documentation (cont)

• DODS Java Programming Reference (JavaDocs) -
  Documentation for the OPeNDAP Java classes
  including the core classes and the SQL server
  classes.
• DODS SQL Server Quick Setup Guide - A simple
  step-by-step guide to setting up the OPeNDAP SQL
  server.
• DODS Matlab GUI User Manual - Describes the
  OPeNDAP Matlab client and GUI.
• DODS FreeForm Server User Manual - Describes the
  OPeNDAP FreeForm server.

101
OPeNDAP Design Documentation

• DODS Data Access Protocol (DAP) - A technical
  description of the OPeNDAP Data Access Protocol
  (DAP).
• DODS Java Design Documents - An assortment of
  design documents on the OPeNDAP Java code.
• Data Delivery Architecture - An overview of the
  architecture of the communication between the
  OPeNDAP client and server.
• Data Delivery Design - The design for
  communication between the OPeNDAP client and
  server.

102
OPeNDAP Design Documentation (cont)

• Uniform Resource Locators - The specification for
  the use and extension of URLs in the OPeNDAP
  system.
• Data Access Protocol - The methodology for
  exchanging data between the OPeNDAP client and
  server.

103
http//opendap.org http//nvods.orghttp//unid
ata.ucar.edu/packages/dods