The GFDL Data Portal A NOAA partnership

1
The GFDL Data PortalA NOAA partnership

• Kevin OBrien
• Steve Hankin
• Serguei Nikonov
• V. Balaji

2
Whats in store

• Background on PMEL/GFDL partnership
• Introduction to Live Access Server
• Technical challenges
• How LAS, OPeNDAP, Unidata tools allow us to face
  technical challenges

3
Background

• GFDL
• Research Lab in New Jersey
• Products include Model ouput and metadata
• IPCC participating member
• Need to expose model outputs to customers

4
Background

• GFDL

5
Background

• PMEL and TMAP
• PMEL is a research Lab in Seattle
• TMAP group at PMEL
• Developed Ferret
• Developed Live Access Sever (LAS)
• Integrate Ferret, LAS and other tools into GFDL
  data portal
• Study interannual variations of the tropical
  ocean-atmosphere systems (ie ENSO)

6
Background

• PMEL

• TMAP

7
Whats in store

• Background on PMEL and GFDL partnership
• Introduction to Live Access Server
• Technical challenges
• Using LAS, OPeNDAP, Unidata tools to overcome
  technical challenges

8
The Live Access Server (LAS)
Introduction to LAS
A highly configurable Web server designed to
provide flexible access to geo-referenced
scientific data
9
LASTwo types of configurability
Introduction to LAS

• Configurable hierarchical presentation
• to address different information architectures
• Configurable data browse/access
• to address grids, variables, constraints,

10
Introduction to LAS
Configurable metadata access
Example NVODS server
11
Introduction to LAS
Configurable hierarchy of information
Example GODEA Intercomparison
12
Introduction to LAS
Configurable hierarchy of information
Example NDBC Buoy Prototype
13
Introduction to LAS

• Once the dataset or variable of interest has been
  identified

14
Introduction to LAS
15
Matlab, IDL, IDV, Ferret, GrADS,
Information Products
netCDF, HYCOM binary, ASCII, GIS layers
16
Introduction to LAS
17
FDS (Ferret Data Server)

• FDS is an OPeNDAP server
• Automatically installed with LAS
• Therefore every gridded LAS dataset is served via
  OPeNDAP
• Has extended analysis capabilities beyond basic
  OPeNDAP servers

18
Introduction to LAS
Compare Fields
19
Introduction to LAS
Compute simple analyses
20
Introduction to LAS
LAS
UI
LAS core
Meta-
Internet
(perl)
data
Standard
legacy
Local
application
Data
Batch
(
Ferret
)
K-12
Multiple User Interfaces
The UI talks to LAS through an XML web service
21
Introduction to LAS
22
Introduction to LAS
23
Whats in store

• Background on PMEL and GFDL partnership
• Introduction to Live Access Server
• Technical challenges
• Using LAS, OPeNDAP, Unidata tools to overcome
  technical challenges

24
Technical challenges include

• Managing large collection of model files

• Directing customers to desired data

• Controlling resource-intensive data requests

• Dealing with curvilinear coordinate systems

25

• Technical challenges include
• Managing large collection of model files
• Directing customers to desired data
• Controlling resource-intense data requests
• Dealing with complex coordinate systems

26
Technical challenges include

• Managing large collection of model files
• Annual, monthly, daily, 3-hourly, etc

• Directing customers to desired data

Solution Unidata Aggregation server and OPeNDAP

• Handling resource-intensive data requests

• Dealing with curvilinear coordinate systems

27
Technical Challenges
Solution Unidata Aggregation server and OPeNDAP

• Unidata Aggregation server
• Combines a group of files/datasets into one
  logical dataset
• Uses THREDDS catalogs to specify the datasets it
  serves

• OPeNDAP
• Allows access to remote data via the internet,
  using http
• format of the native data not a worry
• only need to transfer the subset of interest

28
Technical challenges include

• Managing large collection of model files

• Directing customers to desired data

• Handling resource-intensive data requests

Solution Use LAS categories

• Dealing with curvilinear coordinate systems

29

• Techinical challenges include
• Managing large collection of model files
• Directing customers to desired data
• Controlling resource-intense data requests
• Dealing with complex coordinate systems

30
Technical Challenges
Solution Use LAS categories

• Create logical paths for data access
• Create links into specific LAS categories
• Can link to categories from outside LAS

31
Customize LAS More on categories
GFDL Data portal
32
LAS categories GFDL Data Portal
33
LAS categories GFDL Data Portal
34
Search
coupled
coupled
35
Index
36
Technical challenges include

• Managing large collection of model files

• Handling resource-intense data requests

• Directing customers to desired data

Solution LAS defaults to a batch mode for
large requests

• Handling resource-intensive data requests

• Dealing with curvilinear coordinate systems

37

• Techinical challenges include
• Managing large collection of model files
• Directing customers to desired data
• Handling resource-intense data requests
  Solution LAS defaults to a batch mode for
  large requests
• Dealing with complex coordinate systems

38
Technical Challenges

• Interactive time out configurable by LAS
  installer
• Large requests get backgrounded automatically

39
LAS Batch mode
40

• Techinical challenges include
• Managing large collection of model files
• Directing customers to desired data
• Controlling resource-intense data requests
• Dealing with curvilinear coordinate systems

41
Technical challenges include

• Managing large collection of model files

• Dealing with curvilinear coordinate systems

• Directing customers to desired data

Solution Ferret, FDS, regridding

• Handling resource-intensive data requests

• Dealing with curvilinear coordinate systems

42

• Need something here to explain what you are
  talking about wrt complex coordinates
• Something like having to deal with both
  rectilinear and curvilinear datasets
• Special needs of products for curviliinear
  datasets
• Curv to rect mapping for xy plots
• Addl variables need for netCDF subsets

43
XY plot of curvilinear data
44
On-the-fly regriddingfrom curvilinearto
rectilinearcoordinates
45

• netCDF subset of curvilinear data
• Using nc operators

46
Future directions

• LAS
• Redesign of product server using Java - no Perl

• GFDL Data Portal
• Integrate Curator metadata system with LAS
• To ease dataset configuration
• Increase curvilinear product offering
• Use LAS categories to unite discontiguous time
  regions
• Add averaging and masking of spatial regions

47
Future directions

• Automation
• Implement standards for regridding
• Expand Modeler community
• Increase fusion of GFDL models with others
• OPeNDAP
• LAS Sister Servers
• Several different LAS servers which share XML

48
For more information

• Ferret http//www.ferret.noaa.gov
• LAS http//www.ferret.noaa.gov/Ferret/LAS
• GFDL data portal http//nomads.gfdl.noaa.gov
• OPeNDAP http//www.opendap.org

Thank you!