A Data Access Framework for ESMF Model Outputs

1
A Data Access Framework for ESMF Model Outputs

• Roland Schweitzer
• Steve Hankin
• Jonathan Callahan
• Kevin OBrien
• Ansley Manke

2
Some Motivation

• Model Components produce
• ? Model Runs which produce
• ? Data and Metadata which
• should/could produce
• ? A data portal which allows user to
• Search, browse, compare, validate and analyze
  results
• ? which in turn helps produce insight

3
Some More Motivation

• ESMF and the ESCurator manage components, runs
  and output
• LAS is a candidate to provide a data portal

4
The Live Access Server (LAS)
A highly configurable Web server designed to
provide flexible access to geo-referenced
scientific data
5
LAS

• LAS is an information broker that
• ingests descriptions of data
• data sets
• variables
• (optional reshuffling of above into new
  categories)
• domains 1d-4d
• special constraints (ensemble index)
• rendering hints (smoothing, palettes)
• presents the descriptions to the user in a
  coherent fashion
• asks other software to perform tasks for the user
• and gives the results to the user

6
Architecture Components
User Interface Server (UIS)
Product Server (PS)
Backend Server (BES)
Backend Server (BES)
Ferret Data (OPeNDAP) Server (FDS)
Data
7
LAS Offers

• Independence from underlying data
• Uniform interface for browse (visualize),
  compare, validate and analyze results for many
  data types
• Easy way to make connections to metadata and
  documentation
• Customizable user interface

8
Example (Multiple Hierarchies)
9
Example (Collapse Hierarchies)
Experimental Development Server
10
Example (Custom Headers)
The NVODS Server (http//www.ferret.noaa.gov/nvods
)
11
Example (Adopt a Drifter)
12
Return to the Motivation

• Model Components produce
• ? Model Runs which produce
• ? Data and Metadata which
• should/could produce
• ? A data portal which allows user to
• Search, browse, compare, validate and analyze
  results
• ? which in turn helps produce insight

13
Data Browsing
14
Analysis and Comparison

• Compute differences
• Can define new variables that are the result of
  some analysis operation (done in delayed mode so
  only data that is needed is computed)

15
Analysis and Comparison

• Can use FDS to define regriddings (or other
  analysis) to allow more complex comparisons

16
Validation

• Use FDS to create samplings of in situ
  observations which can be compared to model data
• (This is a planned for a future release.)

17
Detailed Analysis

• Can jump to local tool (Matlab, IDL, Ferret,
  etc.) by requesting scripts for that application
• Scripts encode access to remote LAS data via FDS
• Can take advantage of FDS analysis to compute
  some data reductions and transfer only the result
  to the local application

18
Additional Flexibility beyond Configuration

• Use LAS framework with different backend engine
  (new Product Server is designed to make this as
  easy as possible)
• Web-services interfaces
• To Product Server from other clients
• Will allow implementation of well-known service
  like WMS

19
Architecture Components
User Interface Server (UIS)
Product Server (PS)
Backend Server (BES)
Backend Server (BES)
Ferret Data (OPeNDAP) Server (FDS)
Data
20
Access Control and the GRID

• The new architecture makes participating as a
  GRID partner easier
• Web services-based authentication schemes can be
  used by the Product Server and Backend Server to
  get authentication information (if not supplied
  by a browser certificate).
• Implemented as another filter in a
  already-existing mechanism for filtering on
  IP-address, number of accesses, etc.
• Configuration metadata can be augmented to
  identify allowed access roles by dataset and/or
  variable (and maybe by output product).

21
Return to the Motivation

• Model Components produce
• ? Model Runs which produce
• ? Data and Metadata which
• should/could produce
• ? A data portal which allows user to
• Search, browse, compare, validate and analyze
  results
• ? which in turn helps produce insight

22
More Information

• www.ferret.noaa.gov
• Subscribe to the LAS Users mailing list

23
The End
24
LAS Architecture with OPeNDAP
browser interface
browser
servers
LAS XML configuration metadata
UIS
UI (java/script)
HTTP Request (later SOAP)
PS
HTTP Response
browser
SOAP
gif,netCDF,etc.
BES
BES
BES
UIS User Interface Server PS Product
Server BES Backend Server FDS Ferret Data
Server
scripts /or commands
products
Legacy OPeNDAP
Legacy OPeNDAP
addXML
OPeNDAP Server
data
25
Architecture Components

• User Interface Server (UIS) keeps browser
  up-to-date with relevant interface widgets
• Product Server (PS) takes users requests and
  returns results
• Backend Server (BES) runs legacy application to
  fulfill request
• Ferret Data Server (FDS) General OPeNDAP server
  for LAS data

26
The UI Server Architecture

• User Interface Server
• Installation process translates LAS configuration
  into User Interface Widgets need to present
  datasets, variables, geo-spatial and time extents
  in HTML
• Allows dataset collection to be easily organized
  into any hierarchies the installer chooses
• Communicates requests to the Product Server and
  returns responses (images, files, data tables,
  etc.) to the Web Browser
• Many other customization features

27
The Product Server

• Product Server
• Has knowledge of the LAS configuration
• Makes requests from one or more Back-end Servers
• Fully implemented Web-services interface will be
  available to client writers
• Can interact with a series of Backend-servers to
  fulfill a single request from the user client
• Back-end server
• Can produce product itself
• or
• Interact with a single legacy application (like
  CDAT or Ferret) to produce product
• Requests are communicated via Request and
  Response XML that is exchanged via SOAP.

28
FDS (Ferret Data Server)

• FDS is an OPeNDAP server
• Automatically installed with LAS
• Therefore every gridded LAS dataset is served via
  OPeNDAP
• Has extended analysis capabilities beyond basic
  OPeNDAP servers