DAPPER: An OPENDAP Server for In-Situ Data

1
DAPPER An OPENDAP Server for In-Situ Data

• Joe Sirott
• Donald Denbo, John Osborne, Willa Zhu
• University of Washington
• PMEL/NOAA

2
What is Dapper?

• Web server that provides distributed access to
  in-situ data via OPeNDAP protocol
• Clients include GrADS, Matlab, NcBrowse, Java
  Ocean Atlas
• Available data
• PMEL EPIC database
• ARGO profiles
• NODC World Ocean Database 2001
• Includes database loader for adding/removing
  profiles from Dapper database

3
What does Dapper do?

• Efficiently aggregates thousands of in-situ data
  files (gt2000000 for WODB2001)
• Handles several in-situ data formats
• Argo GDAC netCDF
• PMEL EPIC netCDF
• MADIS (Meteorological Assimilation Data Ingest
  System) netCDF
• NODC Argo/COARDS netCDF
• Supports time series or profiles
• Supports space/time queries

4
What is OPeNDAP?

• OPeNDAP (Open source Project for a Network Data
  Access Protocol) is a protocol for remotely
  accessing scientific data
• Uses HTTP as transport

5
Dapper OPeNDAP Example (space/time query)

• Client HTTP requesthttp//foo.org/.../dset.cdp.d
  ods?lat,temp,preslatgt0latlt20
• Dapper HTTP response Dataset
• Sequence
• Float32 lat
• Float64 _id
• Sequence
• Float32 pres,temp
• profile
• location -4.01,1, 4.3,26.808,
  100,24.982, ,
• -52, 2, 8.3,3.77,100,3.74
  3,

6
Dapper OPeNDAP Example (single profile)

• Client HTTP requesthttp//foo.org/.../dset.cdp.d
  ods?lat,lon,time,profile_id20
• Dapper HTTP response Dataset
• Sequence
• Float64 time
• Float32 lat,lon
• Sequence
• Float32 pres,temp
• profile
• location 1034140577999,-52.9,310.144
  , 8.3,3.77,9.1,3.77,
  

7
OPeNDAP (cont.)

• Sequences accessible to clients via Java or C
  libraries
• Dapper specific _id field in outer sequence
  allows unique identification of an individual
  profile

8
Dapper architectural overview

• Uses relational database to store metadata and
  coordinate boundaries for each profile/time
  series file
• Data maintained in original format
• All data delivered as OPeNDAP sequences

9
Dapper architecture diagram
OPeNDAP Protocol (HTTP)
Dapper HTTP Server
OPeNDAP client

CORBA (IIOP)

Climate Data Portal


Java netCDF library
MySQL

10
Climate Data Portal (CDP)

• CORBA based
• Metadata in database (MySQL)
• Variables (measurements) in individual netCDF
  files
• Reads EPIC, COARDS, and Argo GDAC netCDF files
• Originally developed by Don Denbo and Willa Zhu

11
Dapper architecture diagram
Dapper HTTP Server
OPeNDAP client

CORBA (IIOP)

Climate Data Portal


Java netCDF library
MySQL

12
Dapper HTTP Server

• Consists of configurable services
• Climate Data Portal service
• netCDF service
• Services route OPeNDAP requests
• Implemented as Java servlet

13
Dapper architecture diagram
Dapper HTTP Server
OPeNDAP client

CORBA (IIOP)

Climate Data Portal


Java netCDF library
MySQL

Database loader
14
Database loader

• Automatically adds or removes profile info from
  database
• Transactional loader database wont be left in
  invalid state
• Uses Hibernate object relational mapping software
• 20-30ms to load one profile

15
Dapper performance

• For World Ocean Database 2001 (2e6 profiles)
• 600 us/profile for coordinate info
• 5 ms to read one profile
• Faster for smaller datasets
• Dont use MySQL default configuration!

16
Dapper clients from PMEL

• ncBrowse (Don Denbo, John Osborne)

17
Dapper clients from PMEL

• Java OceanAtlas (John Osborne)

18
CDFsync

• Rapid synchronization of netCDF in-situ files
  over a network
• Only compressed file differences transmitted
  across network
• Based on rsync, but optimized for netCDF and file
  lists consisting of thousands or millions of
  files
• Available at http//www.epic.noaa.gov/epic/softwar
  e/cdfsync/

19
Current work

• Thin web client for Dapper
• Query and visualization of profile/time series
  data
• Downloads in alternate formats (spreadsheet, GIS,
  gzipped netCDF)
• Based on AJAX (Asynchronous JavaScript and XML)
  like Google Maps or Gmail.

20
More info

• PMEL Dapper Server
• http//www.epic.noaa.gov/dapper
• Downloads, propaganda
• http//www.epic.noaa.gov/epic/software/dapper/
• Joe.Sirott_at_noaa.gov