Implementation through European

About This Presentation

Title:

Implementation through European

Description:

HCMR / Hellenic National Oceanographic Data Center. State of the art ... Hellenic Center for ... Hellenic National Oceanographic Data Center (HNODC) is a ... – PowerPoint PPT presentation

Number of Views:31

Avg rating:3.0/5.0

Slides: 62

Provided by: timb162

Category:

more less

Transcript and Presenter's Notes

Title: Implementation through European

1
State of the art Technologies for GIS Management
Systems

Implementation through European
Community Research Projects

Lykiardopoulos Angelos, Iona Athanasia, Lakes
Vasilis, Balopoulos Efstathios, Kalkavouras
Constantine
Hellenic Center for Marine Research
2
Sections of this presentation

Data management and Web Services
Quality control in HNODC data base
Poseidon system, a short description

3
Data Management and Web Services a Historical
Brief
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial

At the end of 80s and early 90s several
Oceanographic Data centers had created Databases
with mass storage of spatial Information which
concerned Oceanographic research outcomes.
This was en enormous step of digital storage
because big data volumes organized in Database
Systems.

In other words the data centers owned databases
Based on various RDBMS mechanisms
With their own database Schemas
With their own data vocabulary (if existed)
With raw data in various formats
With Applied various processes of data quality
control (or with no data quality control)
This effort leaded into the creation of several
data bases without any Interoperability among
them.

But the oceanographic community had the need of
homogenized and comprehensive data sets for
various regions and even world wide.
Due to the limitations of information
technologies this goal was achieved in the
framework of several E.U. projects
Central accumulation of data from various Data
Centers took place
Data sets were processed in common data formats
and more or less same Q.C. procedures.
Finally the dissemination of these Data sets was
carried out mainly via digital media (CD-ROMs,
etc).

During this process
Diverse Data form a wide range of Data Centers
were gathered.
Passed common homogenization and QC procedures.
And finally some comprehensive Data sets were
produced and disseminated.
But
The produced Data Sets were stand alone products
stored in digital media.
Without any dynamic mechanism of maintenance and
enhancement.
And without any online access.

Thus, the idea of a distributed Oceanographic
Data network among the Data Centers start growing
up.
And since Web technologies were being developed
rapidly, became apparent that the above network
had to be web based.

The main obstacles that came through for this
idea to be fulfilled were
Data bases was in different Schemas and
Technologies specific per Data center so there
was not any common data interchange mechanism.
Data were not homogenized among Data centers.
Data quality and Data vocabulary differed from
Database to Database

At the same time another obstacle was that due to
premature Data Base technology many RDMBS systems
lacked spatial capabilities.
So it was very difficult for the end user
To query the Database with spatial criteria
To represent geographically the spatial
information and produce geospatial products
To work interactive with Geo Data through
interfaces with geospatial capabilities
To combine geospatial data in synthetic data
products
Hence, for a distributed Oceanographic Data
network to be established, the necessity of GIS
enabled Databases became apparent as well.

At the end of 90s most of the common used RDBMS
enriched with spatial capabilities and became
then powerful mechanisms of integration,
exploitation and presentation of spatial
information
Worldwide oceanographic data centers, exploit
this advantage and started upgrading their
databases

But still the luck of interoperability was
unsolved. The spatial databases made researchers
life easier with regard to querying data or
representing graphically geospatial
information.But
they were acting without any interconnection
among the various Data Centers.

PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
12
Summarizing the historical brief throughout
important E.U. projects
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial
SEA DATA NET
MEDATLAS
1994
1998
2002
2006
SEASEARCH
MEDAR/MEDATLAS II
PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.

Efforts in QC
Data homogenization
Centralized management
Distribution with CD ROMs

Enrich existing data sets
Establishment of common protocols in QC and
formatting
Enhance communication facilities (www, ftp,
e-mail etc)

Development of a pan-european network for ocean
and marine data information management.
Improve the exchange, availability and
accessibility of ocean marine data and
information.
Expand the online metadatabases with input of
research institutes.
Improve the online accessibility of Sea-Search
services products.

Develop a standardized distributed system for
managing and disseminating the large and diverse
data sets
Enhance the currently existing infrastructures
with web services

FUTURE The new architecture An example Conclusions
13
Web Services
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial

At the same time with the efforts of the
Oceanographic community, during the last five
years new technologies aroused and became common
known as
Web Services

First of all what is a Web Service?
The hidden idea behind WS is the existence of a
universal messaging mechanism, capable to deliver
and translate messages for diverse and different
systems.
This feature came to open the road for the
solution of interoperability.
If different and diverse systems can interpret
and process requests from each other let them
stay different.

There are various theoretical definitions of
What Web Service is such as
A software system designed to support
interoperable machine-to-machine interaction over
a network. It has an interface described in a
machine - processable format (specifically WSDL).
Other systems interact with the Web service in a
manner prescribed by its description using SOAP
messages, typically conveyed using HTTP with XML
serialization in conjunction with other
Web-related standards. (W3C, 2004)

or
Web services are a distributed computing
architecture. Only this particular architecture
makes use of loosely coupled applications, as
opposed to tightly coupled applications, to
enable applications to communicate. This tightly
coupled concept radically affects how information
systems will work in the future
(Clabby 2003)

Generalizing we may define that
Web services provide a convenient and
standardised way of exposing business logic over
a network (the Internet) by use of specific
components of communication
The basic components of WS are
SOAP (Simple Object Access Protocol)
which provides a Standardized messaging structure
based on XML
UDDI (Universal Description, Discovery and
Integration)
which provides the ability to dynamically select
a service at runtime
WSDL (Web Services Description Language)
which describes how the client communicates with
a Web service or what a service does or where is
located.

In the most simple form a web service
architecture may be depicted as the diagram

The rapid development of technologies towards WS
concept has produced several sets of Web Services
with regard to the area of the WS applicability.
Especially for Geospatial data, Open Geographical
Consortium (OGC) has defined two basic sets of
Geospatial Web Services.

The Web Map Services (WMS) which are concerned
with transforming spatial data into maps
(images).
The Web Feature Services (WFS) which are
concerned with direct access to data - reading,
writing, and updating data.

FUTURE The new architecture An example Conclusions
21
Web Processing Services
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial

Furthermore, more Web services are under
finalization process, one of the most important
of them is WPS (Web Processing Service)
WPS is designed to standardize the way that GIS
calculations are made available to the Internet.
Can describe any calculation (i.e. process)
including all of its inputs and outputs, and
trigger its execution as a Web Service.
Supports simultaneous exposure of processes via
GET, POST, and SOAP, thus allowing the client to
choose the most appropriate interface mechanism.

The HNODC data management case.

FUTURE The new architecture An example Conclusions
23
HNODC
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial

Hellenic National Oceanographic Data Center
(HNODC) is a National Public Oceanographic Data
provider and at the same time a member of the
International Net of Oceanographic Data Centers
(IOC/IODE)
HNODC owns a very big volume of Data and Relevant
information about the Marine Ecosystem

For the efficient management and exploitation of
these Data, the first step was a relational Data
Base to be constructed.
After the necessary stage of data quality control
took place, following the International
standards, as they agreed during several EU
Projects or International Organizations.
Finally a mass volume of over 300.000 station
data concerning , physical, chemical and
biological Oceanographic information, stored in a
Relational Data Base.

The next target aroused then, was an Application
to be constructed capable to represent either
- the Geospatial aspects of this information
- together with the non spatial information
The necessary analysis took place and a
traditional Web Application was developed by the
use of the convenient web tools and
technologies.
Geographical representation was achieved by the
use of a Middle Layer tool (ArcIMS) over the Data
Base Mechanism.

The convenient Multilayer Architecture was
followed
The Relational data base as the back end.
ArcIMS as the Geospatial representation
Mechanism.
Apache and Tomcat in heart of the Middle-Tier
System.
On the top a web Interface with selection
criteria for querying specific data sets in
specific geographical areas.

Although its geospatial capabilities this
Application stranded only as a data geographical
presentation and queering mechanism
without any Data management and Data processing
abilities.
and also without any Data interchanging with
other applications
and with its own data vocabulary

At the same time technical issues aroused,
regarding the Applications performance, due to
the inefficiency of the Geographical mechanism to
handle big volumes of data sets as fast as a Web
Application needs.
For HNODC this approach is today considered as
the fist Version (V1) of an Integrated Web Based
Environment for its Data Management and Data
exploitation.

Thus and at the time that Web Services became a
fact, a second Version was planned
The target of this effort was an Integrated Web
Based environment to be developed, following at
least the basic concepts of a Service Oriented
Application (SOA) which are
The interface contract to any Web Service is
platform-independent.
Any Web Service can be dynamically located and
invoked.
Any Web Service invoked is self-contained. That
is, the service maintains its own state.

For the development, state of the art software
components and tools replaced the old ones
Geospatial and no Spatial Web Services mechanisms
took the place in the Middle- Layer.
Geospatial open source tools and custom
development was employed in presentation layer.

At the same time the application may now
interact with any other SOA application either in
sending or receiving Geospatial Data, since it
inherits the big advantage of interoperability
between Web Services systems.

The Architecture can denoted as follows
At the back End Open source PostgreSQL DBMS (and
not only..) stands as the data storage mechanism
with more than one Data Base Schemas cause of the
separation of the Geospatial Data and the non
Geospatial Data.
UMN Map Server and Geoserver are the mechanisms
for
Represent Geospatial Data via Web Map Service
(WMS)
Querying and Navigating in Geospatial and Meta
Data Information via Web Feature Service (WFS)
And in the near future Transacting and processing
new or existing Geospatial Data via Web
Processing Service (WPS)

WhereGroup MapBender, a geospatial portal site
management software for OGC and OWS
architectures, acts as the integration module
between the Geospatial Mechanisms.
MapBender comes with an embedded data model
capable to manage interfaces for displaying,
navigating and querying OGC compliant web map and
feature services (WMS and transactional WFS).

Apache and Tomcat stand again as the Web Service
middle Layers
Apache Axis2 with its embedded implementation of
the SOAP protocol acts as the No spatial data
Mechanism of Web Services.
(These modules of the platform are still
under development but their implementation will
be fulfilled in the near future.)
Finnaly Web user Interface for the end user
developed based on enhanced and customized
version of a MapBender GUI.

PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
37
HNODC - Architecture
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial
PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
38
HNODC Web Interface
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial
PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
39
HNODC Web Interface
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial
PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
40
HNODC Web Interface
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial
PRESENT Web Services W.S. Architecture Geospatial
W.S. WMS/WFS WPS HNODC General HNODC
Architecture HNODC Interface HNODC W.S.
FUTURE The new architecture An example Conclusions
41
Conclusions
PAST Historical Brief Centralized D.M. From
Centralized to Distributed Model From Non Spatial
to Spatial

As conclusion we may denote that embedding these
new technologies to data management model we can
exploit the advantages to
Inherit interoperability between among Data
Centers and Data Providers
Interchange and combine Data Sets with minimum
programming efforts
Produce comprehensive and self explained new Data
Products combing various and diverse data sources
Help end user to navigate through a wide range of
information

We are happy to say that HNODC joins the global
scientific community by providing and consuming
application Independent data products.

The quality control procedures that HNODC
uses to manage oceanographic data and information
follow the international standards recommended by
IOC, ICES and EU
Documentation of the data sets
Format Conversions (Medatlas)
Quality Control on data and metadata

44
Quality Control in HNODC

Quality Control at Metadata and Data (automatic
and visual)
Format, codes and completeness of information
Cruise and stations headers
Observed data of vertical profiles and time
series of fixed moorings
The results of the QC are added as quality flags
to each numerical value for the location, date,
bottom depth and the data points of vertical
profiles and time series.
The quality controlled data set is archived in
the
HNODCs multidisciplinary database.

45
Quality Control in HNODC

The main software tool deployed for QC is Scoop
Scoop is a open source application aimed in
moderating and visual handling of Quality
Control aspects like
Checks in various data format
Duplication entries in datasets
Visual observation of vertical or time series
profiles
etc

46
Format Checks
47
Automatic Checks of Cruise and Stations headers

Vertical Profiles
duplicates cruises within a preset space-time
radius
date of the stations
ship velocity between two consecutive stations
bottom sounding (ETOPO5)
land position (GEBCO)

Time Series
duplicates entries
Sensor depth check
Series duration check
land position (GEBCO)

48
Automatic Checks of Observed Data

Observed data of vertical profiles and time
series of
fixed moorings

at least two parameters required for the qc
(vertical
reference measurement)
constant profiles or times series (sensor stuck)
data out of regional values
comparison with pre-existing climatological
statistics (LEVITUS, MODB, MEDATLAS). Time series
are compared with internal statistics.
decreasing reference parameter (pressure or time)
pressure must not exceed the bottom depth
spikes
vertical instability (for vertical profiles)

Visual Checks of Cruise and Stations headers

Visual Checks of Observed Data

51
Poseidon System

Poseidon is a system of
Monitoring
Forecasting
Various Information production
For the Greek seas

52
Poseidon System

Poseidon is based on an established network of
observation buoys
And on a specialized operational center for the
processing of the data collected and the
production of forecasts,
POSEIDON system is an infrastructure at the
leading edge of modern oceanography in Europe .
The network of observation buoys records
continuously the physical, biological and
chemical parameters of the Greek seas.
Those data are then transmitted to the
operational center where they are sorted and fed
into forecasting models.

53
Poseidon System

POSEIDON system is a unique planning tool in the
endeavour for the protection of the marine
environment.
It also provides a competitive advantage for the
development of business activity, the prevention
of disaster, and the safeguarding of human life.
In the frame of the world-wide trend for the
development of operational oceanography, POSEIDON
network places Greece among the leading countries
in this field

54
Poseidon System

The system under its full functionality can
provide
immediate (on line) and dependable information
reliable forecast
protection of marine ecosystem
support of marine research

55
The POSEIDON buoy network system
The observation buoys are equipped with sensors
that monitor Air-pressure Air-temperature Win
d speed and direction Wave height, period and
direction Sea surface salinity and
temperature Surface current speed and
direction Sea surface dissolved oxygen Light
attenuation with fluorescence Salinity and
temperature in depths 20-1000 m Chlorophyll-A in
depths 0-100 m Dissolved oxygen in depths 20-100
m Current speed and direction in depths 20-50
m Nutrients Radioactivity
56
Data are transmitted to the Operational Center of
POSEIDON using 3 telecommunications systems

- INMARSAT-C satellite - GPRS - Irridium
57
REAL TIME IN SITU MOORING DATA QUALITY CONTROL
The data pass through some tests
58
QC check
Flag 9 assigned
Missing Data are replaced with 99.999
Sensor-range test The data are tested against
physical limitation of sensors
Flag 4 assigned
pass
Physical-range test The data have to range
within thresholds according to
regional climatology
Flag 2 assigned
Examples
pass
Rate of change test The difference between the
test value with the previous and the next one may
not exceed the timeseries std or a pre-arranged
threshold
Flag 3 assigned
pass
Update Database
Stuck value test The value may not remain
constant compared with a number of previous values
Flag 3 assigned
pass