Global Earth Observation Integrated Data Environment GEOIDE A Short Course

1
Global Earth ObservationIntegrated Data
Environment (GEO-IDE) - A Short Course -
Presentation to
Presenter name and affiliation
2
NOAA Top Ten Challenges1

• 10) Alphabet Soup
• 9) Stove Pipes
• 8) Integration
• 7) Architecture
• 6) Data Sharing

• 5) User Needs
• 4) Maximizing Benefits
• 3) Communication
• 2) Data Management
• 1) Execution

• Vice Admiral Lautenbacher in address to American
  Meteorological Society Annual Meeting, January
  30, 2006

3
Data Management - a top priority

• Improving data management is among the highest
  priority challenges facing NOAA integrated data
  management is at the heart of the GEOSS concept

4
NOAAs GEO-IDE

• Scope NOAA-wide architecture development to
  integrate legacy systems and guide development of
  future NOAA environmental data management systems
• Vision NOAAs GEO-IDE is envisioned as a
  system of systems a framework that provides
  effective and efficient integration of NOAAs
  many quasi-independent systems
• Foundation built upon agreed standards,
  principles and guidelines
• Approach evolution of existing systems into a
  service-oriented architecture
• Result a single system of systems (user
  perspective) to access the data sets needed to
  address significant societal questions

5
Why is improved integration needed? Important
societal issues require data from many observing
systems
Discipline Specific View
Whole System View
Current systems are program specific, focused,
individually efficient. But incompatible, not
integrated, isolated from one another and from
wider environmental community
5
6
Todays Challenges

• Incompatible syntax (formats) and semantics
  (terminology) among science disciplines within
  NOAA. Thousands exist. Several examples
• Naming standards Surface Air Temperature
• Meteorology (WMO) named Temperature/dry bulb
  temperature
• Meteorology (air pollution) named Boundary layer
  temperature
• Oceanography named Air Temperature
• Location standards (latitude, longitude,
  elevation)
• Lat/Lon can be degrees/minutes/seconds or degrees
  to tenths and hundredths
• Latitude E/W, 0-180 positive and negative, or
  0-360 running east or west
• Z used to designate elevation in both atmosphere
  and ocean but positive is up in the atmosphere
  and down in the ocean
• Formats (gt50 formats used within NOAA
  translators and standards needed)
• GRIB, NetCDF, HDF and others used for gridded
  data
• BUFR, NetCDF, and many others used for
  observations

7
Goals

• Through GEO-IDE NOAA will
• Identify and address integration gaps in data
  management systems
• Create interoperability across existing data
  management systems
• Develop and adopt data standards for formats and
  terminology
• Integrate measurements, data, and products
• Examine the need for future data management
  requirements
• and will achieve
• Cost avoidances in NOAA business through improved
  efficiency and reduced duplication
• Reduced risks for US IEOS and GEOSS

8
GEO-IDE - an essential component ofenvironmental
information management for NOAA
Integrated observing, data processing and
information management systems Connected by
NOAAs Integrated Data Environment Contributes
to U.S. Global Earth Observation System (USGEO)
and International Global Earth Observing System
of Systems (GEOSS).
9
Scope

• Concerned with environmental and geospatial data
  and information obtained or generated from
  worldwide sources to support NOAA's mission (as
  defined in NOAA Administrative Order 212-15)
• Does not consider administrative support systems
  such as finance, personnel, acquisition or
  facilities management
• Includes all aspects of data management,
  including data acquisition, ingest, data
  processing, archival and access

10
Vision

• System of systems a framework to effectively
  and efficiently integrate NOAAs many systems
• Minimize impact on legacy systems
• Utilize standards
• Adopt, adapt and only reluctantly create
• Emphasis on flexibility
• Work towards a service-oriented architecture

11
Approach

• Each NOAA LO/program/project continues to manage
  its data independently
• Service Oriented Architecture
• Agreed principles, guidelines and standards

• Reference Federal CIO Council, Jan 06
  "Services and Components Based Architectures A
  Strategic Guide for Implementing Distributed and
  Reusable Components and Services in the Federal
  Government"

12
Service-Oriented Architecture

• Under an SOA, capabilities are built one at a
  time to create Web Services
• Web services can be considered as a layer built
  on top of existing systems in which capabilities
  to access these systems are made available to
  applications that require them
• The fabric of the SOA is built upon standards
  for
• discovery (e.g. CF, FGDC, ISO, SQL)
• transport (e.g. HTTP, FTP, OPeNDAP)
• use (e.g. netCDF, HTML, etc.)
• Can be tightly coupled (SOAP) or loosely coupled
  (REST)

13
SOA Concept
14
SOA (continued)
Four general classes of web services are
anticipated

• Operational Public Access Services for public
  access to data, products and information services
  
• Operational Services where security, timeliness,
  and reliability are paramount
• Scientific Services where efficient and flexible
  discovery and access to data sets are required
• Commercial value-added services

15
Key Development Strategies

• Maintain and minimize impact on legacy systems
• Evolutionary development through pilot projects
• Coordinate activities through Communities of
  Interest organized by Data Types
• Grids, time-series, moving-sensor
  multi-dimensional, profiles, trajectories,
  geospatial framework, point data and metadata

16
Communities of interest based upon data typesAn
initial list

• Grids (e.g. model output, gridded data products)
• Moving-sensor multidimensional fields (e.g.
  satellite swaths, side-scan sonar, weather radar)
• Time series (e.g. fish landings, sun spot
  activity, climate data, paleo-records)
• Profiles (e.g. atmospheric soundings, ocean
  casts, profiling floats)
• Trajectories (e.g. underway ship measurements,
  aircraft track data, ocean surface drifters)
• Geospatial Framework Data (e.g. shorelines, fault
  lines, marine boundaries, map annotations)
• Point data (e.g. tsunami or seismic occurrences,
  geodetic control)
• Metadata - information needed for the use and
  interpretation of data

17
Standards

• NOAA-wide standards are an essential component
• Adopt, adapt and only as a last resort develop
• Process is proposed to guide nomination,
  evaluation and adoption of standards

18
NOAA GEO-IDE Standards Process

• Open clear means for all interested parties to
  participate
• Efficient as streamlined as possible given
  other requirements
• Dynamic standards updated or retired as
  technology and practices evolve
• Coordinated consult with other organizations
  that are facing related standards issues (e.g.
  Federal, GEOSS, WMO, IOOS, industry, etc)
• Technically sound decisions based on technical
  merits and cost/benefits
• Methodical and evolutionary any new standards
  will be harmonized with existing successful
  standards
• Layered adopt broadly accepted industry
  standards but with discipline-specific profiles,
  schemas and vocabularies
• Grounded in software practices standards will
  be tested in functioning software before being
  accepted
• Communication - outreach and education to ensure
  NOAA data managers receive adequate training and
  support in the utilization of standards

19
NOAA GEO-IDE Standards Process(continued)
Phased approach of submission, evaluation and
adoption

• Submitted Standard Standards can be submitted
  by anyone in NOAA for evaluation to see if they
  address needs and could be applicable to NOAA
• Proposed NOAA Standard Can be provisionally
  used within NOAA for technical evaluation
• Recommended NOAA Standard NOAA data systems
  should consider supporting the standard wherever
  applicable for evaluation in real-world NOAA
  systems
• NOAA Standard Approved and mandated where
  appropriate

20
NOAA GEO-IDE Standards Process(continued)

• GEO-IDE standards are inclusive not exclusive
• not intended to stipulate that any standard can
  NOT be utilized in a NOAA data system
• Intended to identify standards that should be
  supported by NOAA information management systems.
  Examples
• NOAA systems must support standard A or B or
• NOAA systems exchanging data as part of a
  service-oriented architecture must support this
  standard or
• NOAA system must be able to ingest data using
  this standard or
• NOAA system must be able to output data in
  standard A or B

21
Types of Standards Needed

• Standard names and terminology
• Metadata standards (content and format)
• e.g. FGDC and ISO 19115 w/ remote sensing
  extensions
• Standard formats for delivery of data/products
• WMO, NetCDF, HDF, GeoTIF, JPEG, etc.
• Web Services Standards
• World Wide Web Consortium
• OGC (Features, Coverage, GML)
• OPeNDAP

22
Fast-track submission
An initial set of well known and/or widely used
standards are being considered as an initial set
of submitted standards
23
Data Management Principles

• Leadership Information is a strategic asset and
  information management must be a key component of
  every environmental program
• Stewardship Data must be collected, produced,
  documented and maintained keeping the
  requirements of all users in mind
• Long-term preservation Observations, data
  products of lasting value, and associated
  metadata must be preserved
• Requirements-driven Providers and users of data
  and products must play an active role in defining
  requirements that drive development and evolution
  of data management systems
• Discovery and access Mechanisms to facilitate
  discovery, delivery, use and interpretation of
  data and products are essential

24
Data Management Principles(continued)

• Standards and practices Effective application of
  standards and best practices contributes to
  development of systems that are interoperable,
  efficient, reliable, scalable and adaptable
• Quality Data, products and information should be
  of quality sufficient to meet the requirements of
  society and to support sound decision making
• Cooperation and coordination Each organization
  can realize the full potential of its data only
  by participating in a global community of
  integrated information management systems
• Security Data, information, and products must be
  preserved and protected from unintended or
  malicious modification, unauthorized use, or
  inadvertent disclosure

25
NOAA Guide on Integrated Information Management

• The primary source for guidance on NOAA
  environmental information systems
• Available on-line and updated frequently
• Contains or provides pointers to
• NOAA and other Federally-mandated data management
  policies
• NOAA-wide standards (in all stages of approval)
• Registry of information management tools and
  software
• Planning template with guidelines and/or
  checklist for creating and implementing NOAA
  project-level data management implementation plans

26
Governance Structure and Program Control

• GEO-IDE Project Office
• NOAA Data Management Integration Team (DMIT)
• Technical implementation teams
• Standards Team
• Training Team
• SOA Team
• Communication Team
• And others

27
Advantages for developers

• Single reference point for NOAA data management
  guidelines, principles and standards (The
  Guide)
• Catalog of software tools
• More efficient development by enhanced re-use of
  code and utilization of services
• More customers for your data and products

28
Advantages for scientistsand other customers

• NOAA data and products are easier to
• Locate
• Acquire and load into their favorite software
• Use and understand
• Integrate with other data from NOAA
• Integrate with data from other US-GEO and GEOSS
  partners

29
How you can participate

• Participate in the NOAA standards process
• Nominate standards
• Apply relevant standards to your work
• Develop and distribute any tools, extensions, or
  enhancements to standards that are needed to meet
  your requirements
• Provide evaluation comments to GEO-IDE Project
  Office
• Contribute to the work of one of the GEO-IDE
  implementation teams
• Develop proof of concept prototypes to explore
  what core Web services are needed
• Use core NOAA Web services when developing new
  systems
• Develop Web Service interfaces to your legacy
  systems
• Contribute software to be listed in The Guide

30
Backup Slides