Outcome of the EU review of the DataGRID Project

1
Outcome of the EU review of theDataGRID Project

• Peter Kunszt
• WP2 Manager
• Peter.Kunszt_at_cern.ch

2
Outline

• Review Agenda
• The Result
• The Review
• Introduction and Achievements
• First year objectives
• Architecture overview, issues and actions
• Job submission example (DEMO)
• Plans for 2002 and Issues
• Summary
• The Reactions
• Paris Conference

3
Review Agenda Friday 1st March

• Introduction Kyriakos Baxevanidis/EU Hans
  Hoffman/CERN
• Project overview Fabrizio Gagliardi
• Architecture, middleware delivery schedule
  Bob Jones
• Applications
• WP8 Federico Carminati
• WP9 Julian Linford
• WP10 Johan Montagnat
• Testbed 1 Charles Loomis
• Demos
• WP8 Eric Van Herwijnen Joel Closier
• WP9 Johan Van De Vegte Christine Leroy
• WP10 Vincent Breton Yannick Legre
• Quality Assurance Gabriel Zaquine
• Dissemination Roberto Puccinelli

4
The Result
5
Review Feedback Positive Comments

• Comments by Dr Karsten Decker
• Project is congratulated for exceeding
  expectations and 1st year objectives it has
  been well managed and is on track
• The project management and the project office
  were especially congratulated for an excellent
  job
• All deliverables have been delivered on time
  before the review and they are accepted as is a
  1st for the reviewers
• Reviewers welcome incremental releases this
  will improve testing and feedback
• Demo was useful and realistic not everything
  worked to perfection

6
Review Feedback Constructive Critique

• Must publish results widely in conferences and
  journals - ensure EDG innovations are correctly
  credited to the project and EU
• Licensing issues are important - urged to solve
  this quickly
• Data mgmt should be extended beyond simple read
  facilities to make it attractive to other user
  groups
• In future reviews will look for details of
  scalability, fault tolerance security
• Dissemination needs quantitative results for next
  review e.g. number of contacts made at each
  event
• Some partners overspending need to be understood
  in detail.

7
Project Review - actions

• Extra-ordinary review to be held in May/June at
  ESA site written details of funding activity
  requested
• Written report from reviewers will be produced in
  3 weeks
• Kyriakos Baxevanidis congratulations to the
  projectplease take on-board the recommendations
  from the reviewers

8
The Review
9
Introduction

• Difficult to review a project of this size and
  complexity in one day
• Assume all details in the deliverables and
  periodic reports
• Decision to present the project from the
  applications angle
• Coherent with the original mission of
  demonstrator and test bed for high performance
  research networks (RN Geant)
• Innovative middleware and fabric developments
  presented in a practical demo

10
Achievements of the project

• The project is up and running!
• All 21 partners are now contributing at
  contractual level (took much time and effort)
• Project administrative and managerial structure
  established with minimum resources
• Important survey of existing technology, market
  survey and state-of-the-art performed by most WPs
• Globus selected as basis for the middleware
  (excellent relations established with Globus
  developers in US, stress test of new Globus
  releases)
• First architecture defined, focused on test bed 1
• Necessary middleware developed by WP1-5, adopted
  for future LHC computing project
• Packaging, integration and deployment WP6 with
  essential contribution by application WPs (8-10)
  and support by network and security WP7
• User requirements, test use cases and validation
  provided by application WPs (8-10) Ready to
  deploy real production for D0 and Babar PP
  experiments
• Important dissemination activity and quality
  control provided by WP11
• Overall management and administration deployed by
  WP12

11
Objectives for first year of project

• WP1
• Job resource specification, description
  scheduling
• WP2
• Data access, migration replication
• WP3
• Monitoring infrastructure, directory services
  presentation tools
• WP4
• Framework for fabric configuration management
  automatic sw installation
• WP5
• Common interface for Mass Storage Systems
• WP7
• Network services and monitoring

• Collect requirements for middleware
• Take into account requirements from application
  groups
• Survey current technology
• For all middleware
• Core Services testbed
• Testbed 0 Globus (no middleware)
• First Grid testbed release
• Testbed 1 first release of middleware

12
DataGrid Architecture
13
Architecture Issues and Actions

• Some concepts remain vague
• e.g. what do we really mean by interactive jobs?
• Some boundaries are unclear
• Between components and between work-packages
• e.g. are we certain about the scope/functionality
  of a Storage Element?
• Some requirements are not yet addressed
• e.g. anonymous users
• The various software components are not yet fully
  integrated
• e.g. do the Storage Elements, Computing Elements
  Information System work in unison?
• Short term/ Long term trade-offs
• Need to satisfy the short-term needs of the
  testbed without jeopardizing future directions
• e.g. do we patch code now that we know will be
  replaced in the next release?

• The reformed architecture group will address
  these points taking into account our experience
  from testbed1 and further requirements
• Implementation of iterative releases and
  separation of development testbed from production
  testbed
• Better sw testing and nightly integration

14
A Job Submission Example
Replica Catalogue
Information Service
Resource Broker
Storage Element
Logging Book-keeping
Job Submission Service
Compute Element
15
Summary

• Application groups requirements defined and
  analysed
• Excellent working relationships established with
  the application groups (real users) which are
  driving forward the overall architecture and
  quality of the software produced
• Extensive survey of relevant technologies
  completed and used as a basis for EDG
  developments
• Comprehensive architecture defined as a basis for
  testbed 1
• First release of the testbed successfully
  deployed
• Path for extensions/improvements defined taking
  into account feedback from previous releases and
  needs of the application groups
• Excellent collaborative environment developed
  with key players in Grid arena
• Globus EDG is influencing and participating to
  the extensions of the toolkit to ensure it
  addresses the needs of the application groups
• Other Grid projects Cross-fertilization of
  software components with other Grid projects
  (PPDG/GriPhyN)
• Project can be judged by
• level of "buy-in" by the application groups
• wide-spread usage of EDG software
• number and quality of EDG sw releases
• positive influence on developments of GGF
  standards Globus toolkit

16
Issues

• The project is under-funded relative to the size,
  goals and number of partners
• New technology, 21 partners and diversity of
  applications and communities (CS, PP, EO and Bio)
• Original funding assumptions of CERN (and some
  other partners) no longer valid
• Project office under staffed and below minimum
  operating budget one project secretary, one
  technical coordinator and one quality engineer
  marginally funded (0 travel budget and personnel
  cost higher than expected)
• Overall the project within budget, but ½ of
  partners overspent travel and 2 largely overspent
  first year budget
• Almost completely unfunded resources for test bed
  equipment, networking and security

17
Plans (Technical) for 2002

• Extension of testbed
• more users, sites nodes-per-site
• Iterative releases up to testbed 2
• incrementally extend functionality provided via
  each Work Package
• better integrate the components
• improve stability
• Testbed 2 (fall 2002)
• Anticipate impact of OGSA
• Ian Foster Carl Kesselman members of reformed
  architecture group
• Ensure convergence with US Grid project
  activities (PPDG/GriPhyN)
• to be addressed via InterGrid DataTAG activities

• Planned intermediate release schedule
• TestBed 1 November 2001
• Release 1.1 January 2002
• Release 1.2 March 2002
• Release 1.3 May 2002
• Release 1.4 July 2002
• TestBed 2 September 2002
• Similar schedule will be organised for 2003
• Each release includes
• feedback from use of previous release by
  application groups
• planned improvements/extension by middle-ware WPs
• use of software infrastructure
• feeds into architecture group

18
Future Plans (Administrative/Financial)

• Restructure coordination budget (convert
  overheads in travel, convert part of IR Forum in
  Project Office administrative staff)
• Merge project management and project office with
  DataTAG and integrate dissemination and technical
  coordination with GridSTART
• Economy of scale, synergy of activity
• Explore new unfunded resources PP LCG, national
  Grids (GridPP, INFN Grid, French research Grid,
  etc.)
• Prepare larger follow-up proposal for FP6

19
The Reactions
20
Review Feedback First Reactions

• Must publish results widely in conferences and
  journals - ensure EDG innovations are correctly
  credited to the project and EU
• Licensing issues are important - urged to solve
  this quickly
• Data mgmt should be extended beyond simple read
  facilities to make it attractive to other user
  groups
• In future reviews will look for details of
  scalability, fault tolerance security
• Dissemination needs quantitative results for next
  review e.g. number of contacts made at each
  event
• Some partners overspending need to be understood
  in detail.

Papers in preparation for HPDC, GGF, SC2002,
iGrid, grid2002
Agreed on BSD-like licensing model in Paris.
The update mechanisms are in the design phase and
on track.
Security Group and Architecture Group will deal
with these Issues in great detail
Mechanisms are being set up
Review of ESA in June
21
Paris DataGrid Conference

• Public Industry Research Forum organised by the
  French Ministry of Research high visibility and
  publicity
• WP coordination and planning sessions in parallel
• Interoperability
• Reactions to review
• Plenary reports
• Status of each WP
• Architecture Group
• Security Group
• Other Grid Projects and relationships