PanEuropean infrastructure for marine and ocean data management

1

• Pan-European infrastructure for marine and ocean
  data management
• Unified access to distributed data centres
• By
• Dick M.A. Schaap Technical Coordinator
• Brussels, MODEG Meeting, December 08

Supported by the European Commission FP6 -
Research Infrastructures
2
SeaDataNet Infrastructure objectives

• SeaDataNet is an IR project (2006 2011) for
  further development and operation of an efficient
  Pan-European distributed infrastructure for
  managing marine and ocean data by connecting
• 40 National Oceanographic Data Centres (NODCs),
  national oceanographic focal points, and ocean
  satellite data centres, in Europe
• these Data Centres are mostly divisions of major
  national marine research institutes and based in
  35 countries, surrounding the European seas
• The NODCs operate a national network in many
  countries, especially for data acquired via
  research vessels
• SeaDataNet aims to ensure a comparable quality of
  data sets and to make data sets easily accessible
  on-line through a unique portal, while the data
  sets are stored and managed at the Data Centres.

3
SeaDataNet Participating Countries
49 Partners 40 Data Centres ICES IOC-IODE
JRC specialists in statistics, products and
informatics
4
SeaDataNet Infrastructure overall concept
5
SeaDataNet Infrastructure versions and planning

• Version 0 Continuation and maintenance of
  existing Sea-Search systems and population of the
  Common Data Index (CDI) V0 system
• Version 1 Harmonised and upgraded metadatabases
  adopting ISO 19115 standard Transparent data
  access involving all Data Centres via
  implementation of Common Data Index (CDI) V1
  system and downloading services.
• Version 2 Adding OGC viewing services and
  further virtualisation of data access.

6
SeaDataNet Version 0

• Set-up of SeaDataNet portal (www.seadatanet.org)
• Continuation of existing Sea-Search metadata
  systems
• EDMED - European Directory of Marine
  Environmental Data Sets
• EDMERP European Directory of Marine Research
  Projects
• CSR Cruise Summary Reports
• EDIOS European Directory of Ocean observing
  Systems
• EDMO European Directory of Marine Organisations
• Upgrading of the Common Data Index (CDI) pilot
  and population of CDI V0 by all Data Centres
• Operation of existing data access systems /
  procedures of all Data Centres, included in the
  CDI V0 directory

7
SeaDataNet Portal website
www.seadatanet.org
8
SeaDataNet Common Data Index V0

• CDI is a fine-grained index to individual data
  measurements (such as a CTD cast or moored
  instrument record). This is in contrast to EDMED,
  which is an inventory of data sets, each of which
  comprises a large number of measurements.
• CDI paves the way for direct online data access
  through file downloads or online data requests.
• CDI XML Schema is based upon the ISO 19115
  standard
• CDI metadata contains standardised information
  about data access restrictions
• Present CDI V0 system gt 340.000 CDI entries from
  36 Data Centres from 29 countries

9
SeaDataNet CDI V0 User Interface
10
SeaDataNet CDI V0 Data coverage points
11
SeaDataNet CDI V0 Data coverage - examples
12
SeaDataNet CDI V0 Data Access set-up

• Central and homogeneous metadatabase, but for
  Data Access users are confronted with the
  different user interfaces of the SeaDataNet Data
  Centres (which can provide online search and
  access / ordering facility / e-mail request)

13
SeaDataNet CDI V0 Data Access example
RIHMI-WDC Russia
14
SeaDataNet System Approach for V1 and V2

• An approach has been adopted, which is in line
  with INSPIRE.
• The SeaDataNet infrastructure should consist of
  the following services
• Discovery services Metadata directories
• Security services Authentication, Authorization
  Accounting (AAA)
• Delivery services Data access downloading of
  data sets
• Viewing services Visualisation of metadata,
  data and data products
• Product services Generic and standard products
• Monitoring services Statistics on usage and
  performance of the system
• Maintenance services Updating of metadata by
  Data Centres
• A network of interconnected Data Centres and a
  central Portal, that will give users access to
  the various SeaDataNet services, and information
  on data management standards, tools and protocols.

15
SeaDataNet Users portal architecture V1
Data download
User registration
Status of request
Data request
General request
Metadata request
data
metadata
Project info
Registr.
Standards
Cross search
My transact.
Requestsstatus manager
CDI
Shopping basket
EDIOS
EDMED
Software
AAA
CSR
EDMERP
EDMO
Vocabularies
Organisation data source id
User Register
CSR
EDIOS
CDI
EDMO
BODC Database
EDMED
EDMERP
BSH Database
at BODC
at BSH
at MARIS
Metadata Data catalogues
...
Entry point for access hits
services for metadata
Data downloading services
Download managers in Data centers
Visualization services (WMS) for V2
16
SeaDataNet Interoperability

• Interoperability is the key to distributed data
  management system success. This is achieved in
  SeaDataNet via
• Using common quality control protocols and flag
  scale
• Using common and controlled vocabularies,
  including international content governance
• Adopting the ISO 19115 metadata standard for all
  metadata directories
• Providing XML Validation Services to quality
  control the metadata maintenance
• Providing standard metadata entry tools (MIKADO
  CMS)
• Using harmonised Data Transport Formats (NetCDF,
  ODV ASCII and MedAtlas ASCII) for data sets
  delivery
• Adopting of OGC standards for mapping and viewing
  services
• Using SOAP Web Services in the SeaDataNet
  architecture

17
SeaDataNet Quality Control Guideline

• A guideline (V1) of recommended QC procedures,
  reviewing NODC schemes and other known schemes
  (e.g. WGMDM guidelines, World Ocean Database,
  GTSPP, Argo, WOCE, QARTOD, ESEAS,SIMORC, etc.).
• QC methods for CTD (temperature and salinity),
  Current meter data (including ADCP), Wave data,
  Sea level data
• A scheme of QC flags to be used in
  SeaDataNet.These flags are for assigning to
  individual data values. They are not for
  allocating to whole data series, or to
  accompanying information.
• Compiled in discussion with IOC, ICES and JCOM,
  to ensure an international acceptance and tuning.
  Important feedback from the joint IODE/JCOMM
  Forum on Oceanographic Data Management and
  Exchange Standards (January 2008), joined by
  SeaDataNet and international experts to consider
  on-going work on standards and to seek
  harmonisation, where possible.
• Now extending the guideline with QC methods for
  surface underway data, nutrients, geophysical
  data, and biological data. V2 April 2009.

18
SeaDataNet Common Vocabularies

• Use of common vocabularies in all metadatabases
  and data formats is an important prerequisite
  towards consistency and interoperability.
• Set-up and population of Common Vocabularies. The
  SeaDataNet Vocabulary service is based upon the
  NERC DataGrid (NDG) vocabulary Web service. For
  end-users there is a vocabulary Client Interface
  for searching and browsing and to export selected
  entries in csv format.
• The Web service is compliant to WS Basis Profile
  1.1, which is adopted as standard for all Web
  services in SeaDataNet.
• Content governance of the vocabularies is very
  important and is done by a combined SeaDataNet
  and MarineXML Vocabulary Content Governance Group
  (SeaVoX), moderated by BODC, and including
  experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS
  and more international groups. SeaVox operates by
  mailing list server.

19
SeaDataNet Common Vocabularies
Vocabularies User Client Interface
http//seadatanet.maris2.nl/v_bodc_vocab/welcome.a
spx
20
SeaDataNet Common Data Transport Formats

• V1 data sets are accessable by downloading
  services. Delivery of data sets to users requires
  common data transport formats, which interact
  with other SeaDataNet standards (Vocabularies,
  Quality Flag Scale) and analysis presentation
  tools (ODV, DIVA)
• The following formats have been defined
• SeaDataNet ODV4 ASCII for profiles, time series
  and trajectories
• SeaDataNet MedAtlas as optional extra format.
• NetCDF with CF compliance for gridded data sets
• ODV4 and MedAtlas have been outfitted with a
  SeaDataNet semantic header
• International cooperation is underway from
  SeaDataNet with the CF community and UNIDATA for
  a common NetCDF format (Core Data Model CDM)
  for the oceanographic and meteorological domains,
  including a semantic header

21
Authentication, Authorization and Administration

• Metadata systems and website with standards and
  tools are public domain
• Single Sign On system required for access to
  distributed databases
• User must register once in order to get a
  personal login - password
• Web form to provide necessary information
• User agrees with SeaDataNet User Licence
• After processing, login/password sent by email
  (email check)
• Licence is part of the SeaDataNet Data Policy,
  that is intended to be fully compatible with the
  Directive of the European Parliament and of the
  Council on public access to environmental
  information, the INSPIRE Directive, IOC, ICES,
  WMO, GCOS, GEOSS and CLIVAR data principles.
• NODCs intermediate in the registration of their
  national users to the SeaDataNet central user
  register

22
SeaDataNet V1 Discovery Services

• Discovery services
• EDMED - Data Sets
• EDMERP Research Projects
• CSR Cruise Summary Reports
• EDIOS Monitoring systems
• EDMO Marine Organisations
• CDI Common Data Index
• gateway to data sets

23
SeaDataNet V1 Discovery services

• Activities undertaken for
• Reviewing and streamlining the logical formats of
  each of the Directories
• Expanding the number of Common vocabularies,
  further population and upgrading of Vocabularies
  Web services
• Defining XML schemas and formats, using the ISO
  19115 metadata standard as basis
• Defining and developing maintenance modalities
  for each of the Directories

24
SeaDataNet Discovery services

• Defining and developing new User Interfaces for
  each of the Directories
• Defining and implementing XML Validation Web
  services, that will be used to validate XML
  output from data centres, before import into the
  public Directories
• Developing Web services for the Directories

25
SeaDataNet new User Interfaces
EDMERP
CSR
EDIOS
EDMO
26
SeaDataNet Delivery Services CDI V1

• Central and homogeneous metadatabase plus
  transparent access to the distributed data sets
  via a unique user interface at the SeaDataNet
  portal and downloading services

27
SeaDataNet CDI V1 system components

• CDI V1 Metadatabase and User Interface
• Searching and browsing of metadata of data sets,
  managed by SeaDataNet Data Centres
• Central User Register - Web service
• Details of users, their organizations and
  addresses, license period, Id-Passwords,
  SeaDataNet Roles
• Shopping Basket User Interface
• Preparing a user request of multiple data sets,
  handling validation of users and roles via the
  Central User Register, and routing requests to
  the Request Status Manager
• Request Status Manager Web service User
  Interface
• Processing and administration of all requests and
  data deliveries (downloads), for users in
  communication with data centres. For data
  centres to oversee all transactions.

28
SeaDataNet CDI V1 system components

• Data Centre Download Manager Java Tool
• Handles communication with Request Status Manager
  and takes care that requested files are made
  ready for downloading by User (if OK) in the
  agreed formats via a local website address at the
  Data Centre

29
SeaDataNet Downloading specifications

• Retrieving data sets from
• file management systems
• database systems (e.g. Oracle, MySQL, MS-SQL,
  Sybase, .)
• Export of data files in SDN ODV4 format and
  optional MedATLAS for ASCII type data, or NetCDF
  (CF) for grid type data)
• User has a dynamic personal download webpage

30
SeaDataNet CDI V1 Data Access example
Check Status In RSM
Search
Request Confirmed
Include in Basket
Results
Ready at DC x
Download
Shopping list
Data
SDN format
Submit Authentication
31
SeaDataNet V1 presentation and analysis tools

• Ocean Data View (ODV)
• General data analysis and visualization software
• gt10,000 registered users
• Upgrading ODV software into ODV4 for
• Extending ODVs graphical display capabilities
  and interactive controls for automatic and visual
  quality control and data quality flagging
• Seamless connection to SeaDataNet output SDN
  ODV4 data formats, quality flag scale
• Integration of DIVA gridding software
• Important tool for the SeaDataNet regional Data
  Products
• DIVA software (Data-Interpolating Variational
  Analysis) allows to spatially interpolate
  observations on a regular grid in an optimal way.
  The analysis is performed on a finite element
  grid allowing for a spatial variable resolution
  and a good representation of the coastline and
  isobaths.

32
SeaDataNet V1 presentation and analysis tools
33
SeaDataNet V1 presentation and analysis tools
34
SeaDataNet V1 presentation and analysis tools
35
SeaDataNet CDI V1 - status

• CDI V1 database, interface transaction system -
  Full system tested and operational for a selected
  number of Data Centres
• 1st Group of Data Centres (ca 10 15) well
  underway with configuring their infrastructure
  for connecting to the V1 data system
• All Data Centres will be present in the CDI V1
  system by mid 2009
• This will provide online access to ca. 400.000
  data sets from 40 Data Centres from 35 countries
• Bias in physical oceanography and chemical data
  also some coverage of biological, geological,
  geophysical and hydrographic data.

36
SeaDataNet cooperation and expansion

• SeaDataNet V1 standards and tools will be
  adopted, starting 1st January 2009, by new
  projects
• Upgrade Black Sea SCENE network, (2009 2011)
  involving 41 institutes from the 6 Black Sea
  countries, providing index and access to their
  data and strengthening their national NODC
  networks
• CASPINFO network, (2009 2011) involving 12
  institutes and private industry from the Caspian
  Sea region, providing index and access to their
  data and building their capacity for data
  management and user services
• EuroGeoSurveys members (24 geological and
  geophysical data centres from 16 countries) have
  agreed to adopt the SeaDataNet V1 standards and
  tools and together a proposal Geo-Seas has been
  submitted, which awaits its final stage of
  evaluation.

37
SeaDataNet cooperation and expansion

• SeaDataNet V1 standards and tools will be adopted
  by the online forecasting projects ECOOP and
  MyOcean for providing index and access also to
  real-time and near-real-time data and model
  products
• SeaDataNet has close cooperation and interactions
  with many other international initiatives and
  organisations
• IOC IODE (partner of SeaDataNet)
• ICES (partner of SeaDataNet)
• EuroGOOS SeaDataNet operates the EDIOS
  directory
• EurOBIS and MarBEF will populate CDI V1 for
  giving access to biological datasets.
• MMI USA tuning common vocabularies
• IOOS USA tuning overall set-up
• HUMBOLDT GMES project for ESDI SDN runs ocean
  scenario
• Unidata - netCDF4 development
• GENESI-DR connecting in-situ and satellite data
  centres
• SIMORC metocean data from global oil gas
  industry

38
SeaDataNet INSPIRE implementing rules

• Catalogue Service for the Web CSW ISO AP
  version 2.0.2 (OGC) to enable Discovery Services
• ISO profile 19115 for discovery metadata / 19119
  for services (the Core Metadata model must be
  able to combine metadata from multiple resources
  from different domains)
• SOAP for the bindings (communication)

39
SeaDataNet Multiple resources

• SDN V1 Discovery Services
• CDI, EDMED, CSR, EDIOS, EDMERP
• SDN Generic Product services
• Realize Cross-search INSPIRE compliant CSW
  ISO
• Combine with other relevant ocean and marine
  resources
• THREDDS OPeNDAP NetCDF model data sets
• OGC WMS / WCS servers and maps

40
SeaDataNet Mediator / Brokerage service

• Developed as OS by University of Florence
  (Stefano Nativi)
• Harmonises metadata of resources to ISO 19115
  core
• profile and produces CSW-ISO output

41
Set up of full CDI chain
INSPIRE Community
SDN GI-cat mediator
Core metadata
CDI Metadata
CDI WS
CSW-ISO
SDN Portal GUI
CDI RSM service
DC
Search Find Bind
Data Delivery Services
42
SeaDataNet Conclusions

• SeaDataNet is ready for playing an important role
  in the EMODNET development and operation and must
  be included in the roadmap
• SeaDataNet is based upon NODCs, that are
  embedded in major national institutes and that
  oversee national networks, giving a lot of
  potential for realising EMODNET on a wide scale
  at the working floor
• At present SeaDataNet makes use of European
  research projects for concerting and developing
  its Pan-European infrastructure
• EMODNET can assure an improved long term
  perspective and a more structural foundation for
  the Pan-European cooperation, lifting its profile
  from only research support to also management and
  policy support
• Realising EMODNET is not a technical challenge,
  but moreover an organisational challenge
  building upon and engaging existing initiatives,
  such as SeaDataNet, is vital for its chances to
  succeed. EMODNET must not only develop a top-down
  vision, but also adopt a bottom-up approach.

43
Thank you