The Grid and Meteorology

1
The Grid and Meteorology
Ian Foster Argonne National Lab University of
Chicago Globus Project www.mcs.anl.gov/foster
Image Credit Electronic Visualization Lab, UIC
Meteorology and HPN Workshop, APAN 2003, Busan,
August 26, 2003
2
Overview

• The Grid why and what
• Global knowledge communities
• Resource sharing technologies
• Open standards and software
• The Grid and meteorology
• Opportunities
• Espresso interface
• Earth System Grid project

3
Its Easy to ForgetHow Different 2003 is From
1993

• Enormous quantities of data Petabytes
• For an increasing number of communities, gating
  step is not collection but analysis
• Ubiquitous Internet 100 million hosts
• Collaboration resource sharing the norm
• Ultra-high-speed networks 10 Gb/s
• Global optical networks
• Huge quantities of computing 100 Top/s
• Moores law gives us all supercomputers

4
Consequence The Emergence ofGlobal Knowledge
Communities

• Teams organized around common goals
• Communities Virtual organizations
• With diverse membership capabilities
• Heterogeneity is a strength not a weakness
• And geographic and political distribution
• No location/organization possesses all required
  skills and resources
• Must adapt as a function of the situation
• Adjust membership, reallocate responsibilities,
  renegotiate resources

5
For Example High Energy Physics
6
Grid TechnologiesAddress Key Requirements

• Infrastructure (middleware) for establishing,
  managing, and evolving multi-organizational
  federations
• Dynamic, autonomous, domain independent
• On-demand, ubiquitous access to computing, data,
  and services
• Mechanisms for creating and managing workflow
  within such federations
• New capabilities constructed dynamically and
  transparently from distributed services
• Service-oriented, virtualization

7
The Grid World Current Status

• Substantial number of Grid success stories
• Major projects in science
• Emerging infrastructure deployments
• Growing number of commercial deployments
• Open source Globus Toolkit a de facto standard
  for major protocols services
• Simple protocols APIs for authentication,
  discovery, access, etc. infrastructure
• Large user and developer base
• Multiple commercial support providers
• Global Grid Forum community standards
• Emerging Open Grid Services Architecture

8
What We Can Do Today

• A core set of Grid capabilities are available and
  distributed in good quality form, e.g.
• Globus Toolkit security, discovery, access, data
  movement, etc.
• Condor scheduling, workflow management
• Virtual Data Toolkit, NMI, EDG, etc.
• Deployed at moderate scales
• WorldGrid, TeraGrid, NEESgrid, DOE SG, EDG,
• Usable with some hand holding, e.g.
• US-CMS event prod. O(6) sites, 2 months
• NEESgrid earthquake engineering experiment

9
(No Transcript)
10
NEESgrid Earthquake Engineering Collaboratory
U.Nevada Reno
www.neesgrid.org
11
CMS Event Simulation Production

• Production Run on the Integration Testbed
• Simulate 1.5 million full CMS events for physics
  studies 500 sec per event on 850 MHz processor
• 2 months continuous running across 5 testbed
  sites
• Managed by a single person at the US-CMS Tier 1

12
Key Areas of Concern

• Integration with site operational procedures
• Many challenging issues
• Scalability in multiple dimensions
• Number of sites, resources, users, tasks
• Higher-level services in multiple areas
• Virtual data, policy, collaboration
• Integration with end-user science tools
• Science desktops
• Coordination of international contributions
• Integration with commercial technologies

13
Overview

• The Grid why and what
• Global knowledge communities
• Resource sharing technologies
• Open standards and software
• The Grid and meteorology
• Opportunities
• Espresso interface
• Earth System Grid project

14
The Grid and MeteorologyOpportunities

• Inter-personal collaboration
• E.g., Access Grid, CHEF
• On-demand access to simulation models
• E.g., Espresso
• Access to, and integration of, data sources
• E.g., Earth System Grid
• Dynamic, virtual computing resources
• Metacomputing
• Integration of all of the above
• Collaborative, computationally intensive analysis
  of large quantities of online data

15
Expresso Modeling Interface(Michael Dvorak, John
Taylor)

• Meteorology on demand

16
Earth System Grid (ESG)

• Goal address technical obstacles to the
  sharing analysis of high-volume data from
  advanced earth system models

17
(No Transcript)
18
ESG Strategies

• Move data a minimal amount, keep it close to
  point of origin when possible
• Data access protocols, distributed analysis
• When we must move data, do it fast and with
  minimum human intervention
• Storage Resource Management, fast networks
• Keep track of what we have, particularly whats
  on deep storage
• Metadata and Replica Catalogs
• Harness a federation of sites, web portals
• GT -gt Earth System Grid -gt UltraDataGrid

19
Distributed Data AccessProtocols

• OPeNDAP-g
• Transparency
• Performance
• Security
• Authorization
• (Processing)

Typical Application
Distributed Application
Application
Application
Application
netCDF lib
OPeNDAP Client
ESG client
OPeNDAP Via http
ESG DODS
OPeNDAP Via Grid
data
OpenDAP Server
ESG Server
Data (local)
Data (remote)
Big Data (remote)
20
ESG Metadata Services
21
ESG NcML Core Schema

• XML encoding of metadata (and data) of any
  generic netCDF file
• Objects netCDF, dimension, variable, attribute
• Beta version reference implementation as Java
  Library (www.scd.ucar.edu/vets/luca/netcdf/extract
  _metadata.htm)

ncnetCDFType
ncdimension
ncVariableType
ncattribute
netCDF
ncvariable
ncvalues
nc attribute
22
(No Transcript)
23
Collaborations Relationships

• CCSM Data Management Group
• OPeNDAP/DODS (multi-agency)
• NSF National Science Digital Libraries Program
  (UCAR Unidata THREDDS Project)
• U.K. e-Science and British Atmospheric Data
  Center
• NOAA NOMADS and CEOS-grid
• Earth Science Portal group (multi-agency,
  international)

24
For More Information

• The Globus Project
• www.globus.org
• Earth System Grid
• www.earthsystemgrid.org
• Global Grid Forum
• www.ggf.org
• Background information
• www.mcs.anl.gov/foster
• GlobusWORLD 2004
• www.globusworld.org
• Jan 2023, San Francisco

2nd Edition November 2003