Tools for Accessing Distributed In-situ Data Collections

1
Tools for Accessing Distributed In-situ Data
Collections

• Donald W. Denbo, NOAA/PMEL-University of
  Washington/JISAO
• Jason E. Fabritz, NOAA/PMEL-University of
  Washington/JISAO
• Bernard J. Kilonsky, Sea Level Center/ADCP
  Center, University of Hawaii
• John R. Osborne, OceanAtlas Software
• Nancy Soreide, NOAA/PMEL
• L. Charles Sun, NOAA/National Oceanographic Data
  Center
• Willa H. Zhu, NOAA/PMEL-University of
  Washington/JISAO

2
Challenge

• The process of locating and selecting individual
  observations from many on-line, large collections
  of in-situ data observations is complex
• E.g., millions of individual ocean profiles, time
  series, buoy tracks, etc.

3
Some Answers

• A suite of tools focused on in-situ data and
  metadata
• Locating
• Subsetting
• Graphing, viewing and inter-comparing
• Downloading
• Developed for geographically distributed
  collections of in-situ data
• Freely available

4
Java and Open Source

• Developed in Java
• Tools operate smoothly across a variety of
  hardware platforms and operating systems.
• Reduced development/maintenance costs
• Tools are created from widely-used,
  standards-based, freely available, off-the-shelf,
  Open Source components.

5
Modular components

• SGT Java graphics
• Graphical presentation of oceanographic and
  meteorological data
• NdEdit
• Selecting and subsetting from large collections
  of in-situ data
• XML-based MetaArchitect
• Automated metadata generation
• LDAP
• Network data discovery based on geographic
  region, observed quantities or keyword

6
SGT Java graphics toolkit

• SGT
• A collection of Java classes and interfaces to
  create graphical applications
• Easy-to-use SGT JavaBeans in next release
• Features
• Contour, line and vector plots, user settable or
  automatically scaled axes (including time axes),
  mouse-movable, customizable objects (labels, line
  keys, icons), automatic generation of legends
• Usage
• Wide variety of scientific and commercial
  applications
• 9672 downloads by 3548 unique sites from 64
  countries

7
SGT Javagraphics examples
Two time axis styles
8
NdEditSubsetting observed data from large data
collections

• Interactive, linked windows for viewing or
  selecting individual data observations
• View in 2D slices (lat, lon, depth/height, time)
• Select in box, polygon, section line, along
  bathymetry contours
• Refine data selection by deleting or retaining
  files
• Applications
• Java OceanAtlas (profile data)
• Climate Data Portal (distributed ocean data)
• EPIC (desktop system for in-situ data management)

9
NdEditExplore data locations in linked windows
Ocean profiles viewed in linked windows showing
lat-lon and lat-time views
Built-in or customized bathymetry contourscan be
added to lat-lon view
10
NdEditSelect desired subset from a data
collection
Time window
Filter in time window to view or selecta subset
of the data
11
MetaArchitectXML service for automated metadata
extraction

• Automated scanning of data files to extract FGDC
  compliant metadata to support data discovery
• Flexible and configurable XML-based tool
• Applications
• netCDF data
• NODC data archives
• DODS?

12
Using LDAP for environmental data discovery on
the Internet

• Find data sets on-line based on geographic
  region, observed quantities or keyword
• Automated population of directory directly from
  data servers
• Metadata is instantly known to distributed LDAP
  directories
• Access controls available for experimental data
  products
• Features
• Open Source, standards-based, platform
  independent, scalable, extensible, works with
  application or web client
• Applications
• Climate Data Portal (distributed data access)
• OceanShare (collaborative tool)
• DODS (possible, interested)

13
Applications using the tools

• ncBrowse
• interactive browsing of data and metadata from
  DODS or other netCDF data files
• Climate Data Portal
• centralized access to distributed observing
  system data and historical archives
• Java OceanAtlas
• Intensive browsing of ocean profile data
• OceanShare
• network collaboration tool

14
ncBrowseNetworking data file browser

• Flexible, interactive graphical display of data
  and attributes for wide range of netCDF data file
  conventions
• Reads remote netCDF files
• Reads DODS data files
• Axis mapping, parametric plots, flexible mapping
  of independent variables to axes
• Line, vector and contour plots
• Animations
• Usage
• Data file browse tool for wide range of
  disciplines and programs
• E.g., Unidata, DODS, WOCE data
• 4749 downloads by 2265 unique sites from 55
  countries

15
Climate Data PortalAccess to distributed ocean
observations data

• Data available
• TAO El Nino buoy data (realtime) PMEL
• Global sea level data (realtime) University of
  Hawaii
• Global Temperature-Salinity Profile Program
  (GTSPP) NODC
• PMEL hydrographic data and time series collection
  PMEL
• WOCE Acoustic Doppler Current Profiler data
  University of Hawaii
• Asia Pacific Data Research Center in-situ data
  Intl. Pacific Research Center
• US Coastal Ocean Time Series database NODC
• Future plans WOCE hydrography, WOCE/CLIVAR
  datasets, MEDS Canada, ARGO profiling floats
• Features
• Data discovery services based on
  time/space/keyword (LDAP),
• Graphical data viewing/subsetting (NdEdit), data
  graphics (SGT)
• On-line help, tutorial
• Access from Java application or web client
• DODS interoperability (pending)

16
Java OceanAtlasfor browsing Ocean profile data

• Sophisticated browsing/visualization of profile
  data
• Property-property plots with up to 7 x axes
• Offset property-property profiles (waterfall
  diagrams)
• Contour plots/residual contour plots
• Map plots with optional "bubble" isosurface plots
• Linked browsing of all plot types
• Built-in and custom calculations
• Browse data locations in space/time (NdEdit)
• Select in box, polygon, section line, along
  bathymetry contours
• Refine data selection by deleting or retaining
  files

17
OceanShareNetwork collaboration tool

• Access to data and graphics in a networked
  collaborative tool environment
• Access to local and remote data files, graphics
• Standard collaboration tool features (annotation,
  whiteboard, etc)
• Utilizes CDP, NdEdit, SGT, LDAP, Slide,
  JavaSpaces, Jini, SpaceChat
• Secure Document Repository
• Secure location for sharing
• documents, e.g., proposals, papers in progress,
  etc.
• data, images
• prior collaborative session state
• Access control list for individual documents
  and/or folders
• Document version control
• Documents accessible from web and desktop
• Utilizes WebDAV, Slide, Tomcat servlet container

18
Summary

• Tools
• SGT - Java graphics toolkit
• NdEdit Subsetting ocean observations from
  collections
• MetaArchitect - Automated metadata generation
• LDAP Environmental data directory services
• Applications
• ncBrowse Interactive file browser
• Climate Data Portal Distributed ocean data
  access
• Java OceanAtlas Ocean profile data browsing
• OceanShare Collaboration tool

19
In-situ data tools and applications are freely
available

• Web
• http//www.epic.noaa.gov/epic/software/ep_java.htm
  
• Email
• Donald.W.Denbo_at_noaa.gov