Project: C3-Grid Collaborative Climate Community Data and Processing Grid

1
Project C3-GridCollaborative Climate Community
Data and Processing Grid

• Scheduling in C3-Grid Requirements, Concepts,
  Realization
• Christian Grimme, Alexander Papaspyrou

2
Outline

• Why C3-Grid?
• Applications
• Goals
• Architecture
• Applied technologies
• Scheduling- and data management-aware workflows
• Orchestration of workflow components
• Implementation details

3
Why C3-Grid?

• Applications
• Earth System Science investigates anthropogenous
  influence on climate
• Analysis of simulation data and field
  measurements
• Huge amounts of inhomogenous data distributed
  over various sites
• Goals
• Development of a Grid-based infrastructure for
• Data discovery
• Data processing
• Data analysis

4
Why C3-Grid?
Simulation Data
2.523 1.22E-4 1.548
0.241 1.56E-3 2.801
1.121 2.11E-5 1.231
3.326 1.79E-4 0.918
2.252 1.34E-3 2.238

different format
2.11E-5 1.231
1.79E-4 0.918
1.34E-3 2.238

1.22E-4 2.238
0.93E-4 1.119
2.31E-3 0.258

5
Architecture
6
Applied Technologies

• Gridsphere as framework for portal (UI) services
• Grid core services with Globus Toolkit 4
• Web services and WSRF
• GSI and Shibboleth for security
• JSDL and home-grown Workflow Specification
  Language (WSL)

7
Scheduling- and Data Management-aware Workflows

• Jobs
• Single (atomic) execution units
• Execution in one step
• Input data locally available
• Output data stored locally
• Files
• Datasets in workflow, that
• are created new (output)
• are used for job execution (input)
• possibly reusable

8
Workflow Restrictions
Allowed workflow

• Prohibited elements in a workflow
• Circles, back references
• Conditionals

9
Workflows Orchestration
1
DIS
2b
2a
WSL
Harvester
3
4
Meta- Data
DMS
Scheduler
6
8
7
5
Transfer
WS GRAM
HTTP Daemon
9
Workspace
HH
Köln
Data Provider
10
Details on Scheduling and Data Management
Portal Interface
Scheduler
Info Service
DMS
MDS
D
Sites Programs
virtual
Work- space
Staging Infos Functionality
Interface to Data Provider
RM
RM
RM
Computing Resources
11
WSL Workflow Specification Language