Open Science Grid today

1
Open Science Grid today

• Ruth Pordes, Fermilab
• October 23rd 2005

2
OSG Production
46 CEs, 15459 CPUs 6 SEs http//osg-cat.grid.iu.ed
u/
3
Recent Production Statistics
Running Jobs
500
Queued Jobs
1500
October, 2005
4
What is the scope of Grid Services
ROOT Analysis
Production Scripts
Batch Analysis
Experiment User Applications
Event Data Management Selection
Allocation Accounting
Experiment User Interface Frameworks
Application Middleware
Resource Selection
Workflow Management
Virtual Organization (Physics Group) Administratio
n
System Monitoring, Information Accounting
Common Middleware Services
Data/File Catalogs Data Handling
Job Queues Workload Mgmt
Information Catalogs Repositories
Data Movement Bandwidth Scheduling
Job Scheduling Priority
Monitoring Information Accounting Meters
Security Authorization
Storage Access Management
Grid Middleware Interfaces
A Scalable Performant Global Substrate is Key
Local Network
Resources
Disks Farms Storage Batch Queues Compute
Elements local Storage Elements
Permanent Storage Storage Elements
5
Raising the Bar for Commonality
ROOT Interactive Analysis
Production Scripts
Batch Analysis Framework
Experiment User Applications
Goal for Commonality
Definition of Groups, Roles And Policies
Algorithm framework, Interface to I/O libraries
etc
Definition of datasets, Attributes, Selection
Criteria etc
Group Role, (Resources) Allocation Priority
Workflow/Job Description framework
Workflow definition
Application Middleware
Presentation of Accounting, Monitoring Information
Software builds, packaging distribution
MetaData Infrastructure Dataset Definition,
Selection and Provenance
Allocation Accounting
Experiment User Interface Frameworks
Event Data Management Selection
6
Tension Between

• data intensive science small (open) researcher
  needs
• HENP needs very high performance I/O from WN to a
  Local SE and for data movement between sites.
• Want to lower the threshold of entry to allow
  sites with NSF shared disk areas to participate.
• large VOs and small (open) researcher needs
• VOs develop and deploy services which are not
  needed by simple grid applications.
• keeping robust production infrastructure when
  many new services must be deployed.
• Heterogeneous Infrastructure which means services
  (especially those that affect incompatibilities)
  must be advertised.
• this impacts the architecture what we deploy
  when operations

7
Open Science Grid Release 0.2
User Portal
Submit Host Condor-GGlobus RSL
Catalogs Displays GridCat ACDC MonaLisa
Identity and Roles X509 Certs
Virtual Organization Management
Site Boundary (WAN-LAN)
Compute Element
Monitoring Information GridCat, ACDC MonaLisa,
SiteVerify
Compute Element GT2 GRAM Grid monitor
Storage Element SRM V1.1 GridFTP
CE
Common Space across WN DATA (local SE) APP TMP
WN WN_TMP
PRIMA gPlazma
Batch queue job priority
Authentication MappingGUMS
8
Release 0.4 Focus Area

• Operational Efficiencies
• Grid function
• Higher utilization
• Performance metrics
• not robust
• nor well understood
• OSG growth
• Current operations overhead too high
• Registration human mediated
• Monitoring and response as well
• Managed data movement, cognizant of Network
  characteristcs

9
Release 0.4 Focus Area

• Web Services
• VDT 1.3.7
• incorporates Globus v4
• WS GRAM
• MDS4, service container,
• Deploying now on ITB
• Simplify integration and evaluation of gLite
  components
• Evaluate and understand Clarens
• Begin understanding and evaluation of OGSA
• Workload Management
• Service Catalogs
• Clarens Discovery service
• GridCat
• Job Placement
• Accounting

10
OSG 0.4
User Portal
Submit Host Condor-GGlobus RSL
Catalogs Displays GridCat ACDC MonaLisa
Identity and Roles X509 Certs
Service Discovery
Virtual Organization Management
Site Boundary (WAN-LAN)
Compute Element
Edge Service Framework (XEN) Lifetime Managed VO
Services
Compute Element GT2 GRAM Grid monitor
Storage Element SRM V1.1 GridFTP
GT4 GRAM
CE
Some Sites with Bandwidth Management
Common Space across WN DATA (local SE) APP TMP
Accounting
Full Local SE
Monitoring Information GridCat, ACDC MonaLis
SiteVerify
Job monitoring exit codes reporting
WN WN_TMP
PRIMA gPlazma
Batch queue job priority
Authentication MappingGUMS
GIP BDII network
11
Tensions for OSG

• Have not thought about nor sufficiently agreed
  upon the required metrics and statement of
  value and different implementations are giving
  inconsistent results
• Difficult to get to agreement on boundary between
  required and recommended. Grid-wide
  operations receives only a subset of information.
• Differing policies affect ability to extract
  consistent information. (e.g. account mapping).
• VO focus on Virtual Facility end to end
  monitoring and control means less attention to
  Grid-wide issues. More interest in extending the
  capabilities and capacity.

12
SAMGrid Data handling deployment
2 similar experiments. 2 different data handling
and computing models. Software subject to quite
different stresses.
13
Open Science Grids World
? the gridEGEE, OSG, NorduGridCMS
Datagrid, US CMS Datagrid,US ATLAS Datagrid,
INFN Grid, GridPP, GLOW, FermiGrid, Crimson
Grid, UCLA Grid, GRASE, New York State
GridTeraGrid, Texas GridBrazil and Taiwan and
AU grids
14
Campus Grids Federation
CrimsonGrid
Open Science Grid

GLOW
FermiGrid
15
Inter-Operation Commonality in a Federated World

• Identify the Interfaces and Boundaries.
• Define Policies and Rules of Engagement.
• Understand how to Bridge and Adapt. e.g. BDII
  Filter, Accounting reviews,

Expect to Evolve many Services Most work is in
the future!
Filters Policies
EGEE Communities
OSG Communities
BDII
BDII Network
BDII Network
16
Questions?

• About OSG?
• About OSG futures?
• About OSG Architecture?
• About OSG and Networks?
• ..