European DataGRID for EO

1
European DataGRID for EO

• luigi.fusco_at_esa.int - julian.linford_at_esa.int
• ESRIN, 6-7 May 2002
• CEOS Workshop on GRID

2
Summary

• EO applications and GRID requirements
• ESA EO participation to European GRID projects
  DataGrid
• Ideas for CEOS

3
Earth Observation Community GRID interactive
scenario
Common access to EO missions catalogues Acquisiti
on plan, order, delivery
On demand high level products generation
Parametric data fusion and models integration
Collaborative publishing of results
4
EO and Networking Computing which data models?

• Distributed Computing
• Integration of data from various instruments and
  missions
• High-Throughput Computing
• Interferometry
• On-Demand Computing
• Generation of EO user products
• Data-Intensive Computing
• Archive data re-processing, climate modeling
• Collaborative Computing
• Scientists application interactions, Instrument
  cal/val

Ian Foster and Carl Kesselman, editors, The
Grid Blueprint for a New Computing
Infrastructure, Morgan Kaufmann, 1999
5
High demanding computing
Pomona (Cal) subsidence velocity fields 40
ERS1/2 images (92-99), Ambiguity 28 mm
Digital Elevation Model

• GRID requirements
• large data files (10 GB)
• stages with intensive processing
• science driven value adding

6
Science collaborative environment El Niño
SST
November 1997 El Niño
January 1999 La Niña
SST anomaly
7
Global fire atlas - ATSR 1997
8
Global fire atlas - ATSR 1998
9

• Provide a single access point to space systems to
  emergency rescue organisations in case of
  disasters
• Participating Space Agencies CNES, CSA, ESA,
  ISRO, NOAA,
• Missions RADARSAT ERS, (Envisat) SPOT IRS
  NOAA,

10
Earthnet Facilities real time Infrastructure
SEAWIFS
LANDSAT 7 TERRA/MODIS
AVHRR
SPOT
IRS-P3
MATERA (I)
HISTORICAL ARCHIVES
KIRUNA (S) - ESRANGE
TROMSO (N)
MATERA (I)
STANDARD PRODUCTION CHAINS
MASPALOMAS (E)
NEUSTREL.ITZ (D)
METADATA BROWSE WEB
PRODUCTS
USERS
MULTIMISSION DATABASES FOR REMOTE ACCESS AND
USER SERVICES
ESRIN
USERS
11

• ENVISAT FACILITIES ORGANISATION
• Decentralised architecture, central co-ordination
  and supervision.
• National facilities put at ESAs disposal via
  MOUs and contracts.
• Direct dealing with scientific users (outside ESA
  operational remit)
• Co-operation with value added industry in E.O.
  promotion and in technology transfer from
  research to applications.

LRAC/S-PAC
FIN Co-PAC
UK-PAC
ESOC
F-PAF
D-PAC
F-PAC
ESRIN
I-PAC
E-PAC
12
Stimulating new researchs
AOs Stimulating scientific research world-wide
3500 science Users of ESA data
120 New Cat-1 Projects in 2001
700 Envisat AOs to start in 2002
13
Why GRID in EO? (1)

• EO Community Progressive refinement of data from
  many sourcess to produce higher quality products
• Product generation chain involving distributed
  organisations and users
• Collaborative distributed users and data large
  international cooperation
• Discovery large numbers of products resources
• Interoperabiltiy of catalogue and metadata
  already in operation
• Web based data services

14
Why GRID in EO? (2)

• Massive, non-stop data volumes
• New instruments, sensors product types
• Distributed archives
• Historical dataset reprocessing
• Complex numerical processing algorithms
• Near real-time turnover

15
The Grid from a Services View
Environment
Cosmology
Space Science
Applications
...
S/C modelling
Space weather
EO Environment
Distributed
Data-
Remote
Problem
Remote
Collaborative
Computing
Intensive
Visualization
Solving
Instrumentation
Application
Applications
Applications
Applications
Applications
Applications
Toolkits
Toolkit
Toolkit
Toolkit
Toolkit
Toolkit
Toolkit

E.g.,
Grid Services
Resource-independent and application-independent
services
authentication, authorization, resource
location, resource allocation, events, accounting,
remote data access, information, policy, fault
detection
GRID Middleware

Resource-specific implementations of basic
services
Grid Fabric
E.g., Transport protocols, name servers,
differentiated services, CPU schedulers, public
key
(Resources)
infrastructure, site accounting, directory
service, OS bypass
16
Needed GRID technologies

• Resource-independent and application-independent
  services (middleware)
• authentication, authorization, resource location,
  resource allocation, remote data access,
• accounting, security, quality of services, fault
  detection, real time services,
• Specialized protocols, procedures, data
  standards, operational environments, interfaces
  to EO legacy systems
• EO dedicated portal and user access

17
Participation to GRID initiatives
18
Participation in European GRID projects

• EU funded
• DataGRID Earth Observation application
• EGSO Solar radiance
• DataTAG access to Trans Atlantic Connectivity
• ESA funded
• SpaceGRID vision of GRID systems for space
• ESA internal GRID initiative

19
DataGrid EO application objectives

• Specification of EO requirements
• Bringing Grid-aware application concepts into the
  Earth Science environment
• Adaptation of existing systems and selected EO
  applications to use the DataGrid infrastructure
• Testbed validation through prototyping activity
• Activities handled in coordination and
  synchronisation with other related and relevant
  work packages
• Key partners ESA-ESRIN, KNMI (NL), IPSL (F)
• Associated partners ENEA (I), BADC (UK)

20
GOME Instrument (1 day coverage)

GOMEs Ground track
21
Application of DataGrid in EO

• One Use Case being studied in detail (GOME)
• Develop generic components
• Feedback to DataGrid developers and Architecture
  Group
• Re-use components to add new applications
• Testing in controlled GRID environment
  (ESRIN-ENEA) and in wide-European environment

22
Why Grid in EO?
An Example GOME Use Case Process 1 Year of data
ESA
ESA / KNMI
L1
L2
RAW
L1
Science Application
End User
L2

L3
IPSL
VAL
L2
VAL
Regulated Access to Grid processing power
Secure access to Grid-registered high-volume data
storage
23
DataGrid Overview (1/5)
1. Organization
CVS Repository
Certificate Authorities
User Interface
Replica Management
Computing Element
EDG Rules
Resource Broker
Storage Element
Installation Management
EDG Membership Registration
Information Index
Information Monitoring
Application Environments
Network Monitoring
Documentation
Architecture Group
Integration Testing
Middleware Developers
Middleware Packages
24
DataGrid Overview (2/5)
2. VO registration and information publishing
3. Sites subscribe to one or more VOs
Site A
Grid fabric resources
4. Publish details
25
DataGrid Overview (3/5)
3. Job submission with local data
Certificate Authorities
Information Index
Check certificate
User Interface
Resource Broker
Request status
26
DataGrid Overview (4/5)
4. Data replication
Submit job
User Interface
Replica Manager
Replica Catalog
27
DataGrid Overview (5/5)
5. Job submission using replicated data
28
DataGrid Activities

• Testbed validation
• writing scripts to test and validate Testbed1
  services
• Develop Use Cases for end-to-end GOME processing
  and validation demonstration across three sites
  in Holland, France and Italy)
• Develop EO Grid Application Interfacing
  Components
• for generic application interfacing
• High-speed connection to ENEA HPC network
• Installation of ESRIN DataGrid site
• using DataGrid installation tools
• installation of 2 CEs
• ESRIN cluster using PBS
• ENEA using LSF/AFS
• and 1 SE 0.5TB RAID array on ESRIN cluster
  populated from ESA AMS MSS archive

29
DataGrid Issues

• Very-large-scale, complex system with large
  numbers of participants
• Dealing with new concepts and technology
• Communication and coordination in large,
  distributed, multi-cultural, multi-institutional
  development group
• Agressive deployment of middleware releases
• Driven by needs of HEP
• With EO Biology contributions
• Reliant on HEP making the right choices
• Testbed stability, usability, performance and
  scalability
• Application Grid interfacing layer needs to be
  developed
• After CLIs, need APIs
• Ongoing rapid prototyping and development
• Keeping step with code documentation
• Architecture will evolve according to findings
• Will take time to make fair assessment

30
Future Directions

• In general
• OGSA and integration of Web Services
• Wider uptake of Grid computing concepts
• In EO
• Matrix of common application requirements
• Development of Generic Grid platform interface
  components
• Portals-based
• Application Frameworks

31
Considerations for CEOS involvement in GRID

• gridding of EO emerging technologies and
  services
• Interoperability
• EO data format handling
• Web-mapping
• Archive management
• Demonstrate GRID applications
• International project dimension
• collaborative environment
• relation with IGOS, WGISS Test Facilities
• Support CEOS standardisation approach to
  metadata and data access