Integration through Data Management

1
Integration through Data Management

• The SEACOOS experience
• Accomplishments and Obstacles
• SEACOOS approach to data integration
• Accomplishments and contributions to IOOS
  interoperability
• Some lessons learned
• Coordination with nascent Regional Association

2
SEACOOS was initiated in 2002 with ONR funding
to develop a coastal ocean information system
for FL, GA, SC and NC.
SEACOOS Over-Arching Goal To significantly
increase the quantity and quality of
environmental information from the coastal ocean
of the SE U.S. and facilitate its use in a wide
range of societal, scientific, and educational
applications.
3
Observational Platforms contributed by SEACOOS
partners
4
Real-time observations -- SE Coastal Ocean
5
SEACOOS Modeling coordinating model
simulations for the entire region
6
SEACOOS Information Management

• Data Management Coordinating Committee (DMCC)
• Coordination
• Madilyn Fletcher (USC), Dwayne Porter (USC),
• P.I. representatives
• Ed Kearns (U Miami), Mark Luther (USF), Harvey
  Seim (UNC),
• Nick Shay (U Miami), Jim Nelson (SkIO)
• People who get it done
• Charlton Purvis (USC), Jeremy Cothran (USC),
  Vembu Subramanian (USF),
• Jeff Donovan (USF), Sara Haines (UNC), Jesse
  Cleary (UNC),
• Liz Williams (U Miami), Tom Cook (U Miami)
• Web site documentation/portal
• Chris Calloway (UNC), Claire Eager (UNC)

7
SEACOOS Data Stream On-line Portal for
Near-Real Time Observations
Relational database

uniform data
uniform data
1
University of South Carolina
University of North Carolina
University System of Georgia
Step
raw data
raw data
raw data
raw data
raw data
Aggregation
Courtesy NASA Goddard Space Flight Center

Courtesy Cheryl Burden Ross
Graphics provided by Charlton Purvis,
USC Acknowledgements SEACOOS is a
collaborative, regional program sponsored by the
Office of Naval Research under Award No.
N00014-02-1-0972 and managed by the UNC-Office of
the President.
8
SEACOOS achievements in regional Data Management

• 1. Data commons -- Protocols for data
  providers to make their information available.
  A set of standards for use with a given file
  format
• 2. Methods for data aggregation and display
• Together a demonstration of how an RCOOS can
  function to aggregate and display
  information

9
Data commons

• netCDF adopted as the common data language
• (SEACOOS CDL, currently version 2)
• -- well supported (UCAR, Unidata, Boulder, CO)
• -- supported under DODS/OPeNDAP
• -- interfaces to programming and analysis
  packages
• (e.g., Perl, Matlab)
• -- a flexible system for a variety of
  platforms/sensor types
• (scalar time series, vector time series, vector
  profiler time series, scalar gridded maps, vector
  unstructured maps)

10
Data commons(cont.)

• Data Dictionary
• -- table that registers known standards with each
  other
• -- provides an English language description
• -- down-the-road advantages
• options for data providers flexibility in
  presentation and interpretation of information
• (i.e., if format is registered, the providers
  data can be represented)

11
Data aggregation storage

• Aggregation formats
• netCDF (in situ and remotely-sensed data)
• PNG (some RS imagery)
• Storage
• Relational database (in situ, model output, some
  RS)
• Files (RS imagery)

• Technology
• Perl
• PostgreSQL PostGIS

PostgreSQL PostGIS

12
Normalization

• Reporting time varies
• In-situ data
• E.g. daily, hourly,half-hourly, every 10 minutes
• Remotely-sensed data
• E.g. twice daily
• Reporting area varies
• Remotely-senseddata passes

• Round-the-clock updating is resource intensive
• Balance the server load

13
Visualization graphs animations

• time-series graphs

• animations

Main engines PHP PHP-MapScript Perl
MapServer
14
SEACOOS data aggregation/visualization

• A central aggregation site as opposed to
  distributed network (e.g., LAS)
• Link to software community developing GIS-type
  applications
• Powerful visualization tool without limits to
  numbers of layers

15
Data Dissemination

• OPeNDAP (DODS) data access
• Data sharing with the Open GIS Consortium (OGC)
• Web Mapping Services (WMS) and Web Feature
  Services (WFS)
• WMS -- GIS layers shared without moving the data
• WFS data sub-setting built into url, exchange
  of selected portions of dataset
• SEACOOS role in implementation and
  promotion of
• OGC technology for regional oceanographic
  applications

16
Hurricane/Tropical Storm Jeanne 25 - 27
September, 2004 Merged wind observations and
shore radar
17

• SEACOOS Remotely-Sensed Layers
• WMS-Enabled for Hurricane Charley
• (see OpenIOOS web site -- http//twiki.sura.org/bi
  n/view/Main/CharleyUpgrade)
• modis_ergb_low (USF)
• modis_rgb_low (USF)
• modis_rgb_hi_chesapeake_bay (USF)
• modis_rgb_hi_florida_bay (USF)
• modis_rgb_hi_miss_plume (USF)
• modis_rgb_hi_suwannee_river (USF)
• modis_rgb_hi_tampa_bay (USF)
• oi_sst (USF)
• modis_sst (USF)
• avhrr_sst (USF)
• modis_chl (USF)
• quikscat_wind (JPL)
• met_wind (UNC)
• met_air_pressure_contour (UNC)

18
Some lessons -- Interoperability Demonstrations

• Successfully accomplished (capabilities were
  demonstrated),
• but had fire-drill aspects from
  perspective of technical personnel
• Better coordinated in Summer 2004 than
  previously, a more mature effort
• Better adaptation of OGC WMS and WFS technologies
• Additional layers
• A better product under the hood
• However, the push to get out products in the
  short-term was done at the expense of
  thoroughness
• Process to add new products, identify sources
  seen as somewhat haphazard
• Not the way to build an operational system

19
Other Obstacles / Issues

• QA/QC of real-time data and archives
• Needs further development compliant with
  national standards
• Resource availability and allocation
• Many personnel have multiple roles
• Crucial expertise in a small number of people
• Time for documentation
• Need for redundancy in comm./processing streams
• Data from national providers
• not always in a readily accessible form (often
  requires screen scrapes)
• Engaging users / getting feedback
• In-reach within the regional program as well
  as with external users

20
Other SEACOOS regional DM activities

• Coordination with NDBC, NWS
• e.g., data push from COMPS, SABSOON
• Stations part of NDBC network
• Coordination with developing Regional Association
  (SECOORA)
• e.g., Sensor/equipment inventory
• Development of metadata tool/metadata system for
  the RCOOS (MetaDoor, USC)
• Coordination with State Agencies

21
Equipment Inventory
User Interface Components
Quick Search by variable measured
Query Returns Map
Full Search
Query Returns DB records
22
Equipment Inventory
Admin Interface
This interface provides database access to
equipment managers to add and edit the following
information.

• Institution Institution Name, Abbreviation,
  Observing Group URL, Affiliation
• Contacts Contact Name, Phone Number, Email
  Address, Contact level
• Stations Institutional Station ID, Station Name,
  Lat/Long
• Equipment Package Description, Manufacturer,
  Model, Equipment Type, Vertical Position, Power
  Requirement, Communication Type, Near Real-time
  Status, Variables Measured, Change/Calibration
  History, Active Status, Comments

23
Next Steps
Equipment Inventory

• Data input and involvement from federal data
  providers for a more accurate picture of the
  regional observation system (assets, gaps,
  performance)
• Connecting equipment data as metadata to support
  observations and the QA/QC process. Exploration
  of SensorML (XML) and MetaDoor (metadata tool
  developed at USC).
• Improved data intake capability automated batch
  processing for new equipment
• Expanded interactive map functionality Zoom/Pan