EU DataGrid Project Status and perspective for EU FP6

1
EU DataGrid ProjectStatus and perspective for EU
FP6

• Fabrizio Gagliardi
• Project Leader, CERN
• Fabrizio.Gagliardi_at_cern.ch

2
Outline

• Experience of first year and main issues
• Results of first project review and follow up
• Testbed status
• Relationships with other Grid projects
• Future directions and EU FP6

3
Highlights (from First Project Review on March
1st)

• The project is up and running!
• All 21 partners are now contributing at
  contractual level
• Project administrative and managerial structure
  established with minimum resources
• All EU deliverables (40, gt2000 pages) submitted
  in time for the review according to the contract
  technical annex
• First test bed delivered with real production
  demos

4
Major achievements (from First Project Review on
March 1st)

• Core middleware group developed innovative S/W
  also exported to US (GDMP and resource broker)
• Excellent collaboration with US Globus and Condor
  developments
• Close collaboration with similar US activities
  (PPDG, GriPhyN/iVDGL)
• Large community of enthusiastic, dedicated
  scientists (mostly unfunded)
• End users involved in all stages of the project
  (requirements definition, architecture, S/W
  integration, deployment, validation and tests)
• Unfunded staff effort about twice the EU funded
  (voluntary participation from Portugal, Ireland,
  Russia Denmark both in M/W and in the test bed)

5
more achievements, continued (from First
Project Review on March 1st)

• Good relations to industry (IR Forum and WP11)
• Seed funds for national Grid projects,
  coordinator and initiator of other projects
  (DataTAG, CrossGrid, GridSTART, security
  proposal)
• Initiator and active participant in GGF,
  Intergrid, EIROForum Grid WG, OCDE interest to
  start a WG, exploratory work in Asian Pacific and
  South America
• PM in GGF GFAC, 2 members in GGF WGSG, 2 GGF WG
  chairs
• 2 GGF WGs and 1 GGF research group being proposed
  on networking by WP7
• Pioneering role (EU Grid flagship project) first
  opportunity to work on Grid for ESA with
  fostering effect of internal Grid activity
• Prototype use of national RNs for Grid deployment
  (building Grids of Grids)

6
more achievements, continued(from First Project
Review on March 1st)

• Collaboration with Geant/Dante (WP7 and DataTAG)
• Collaboration agreement established in June 2001
• Technical Proposal
• Test equipment located in three GEANT PoPs
• Analysis of TCP performance over a wide area
  high-speed network
• Demonstration of high-throughput data flows over
  long time scales
• Pilot of the IP Premium service including
  end-to-end tests with biology use-case
  applications (WP10)
• Exploration of use of reliable multicast for file
  replication
• Better understanding of how high-speed
  infrastructures react to traffic load
• Support in the definition of TCP-based services
• Integration and sharing of the existing GEANT
  monitoring with end-to-end DataGRID network
  monitoring data

7
Issues (from First Project Review on March 1st)

• Overall under funded project relative to the
  size, goals and number of partners
• New technology, 21 partners and diversity of
  applications and communities (CS, PP, EO and Bio)
• CERN (and some other partners) original funding
  assumptions no longer valid
• Project office under staffed and below minimum
  operating budget one project secretary, one
  technical coordinator and one quality engineer
  marginally funded (0 travel budget and personnel
  cost higher than expected)
• Overall the project within budget, but ½ of
  partners overspent travel and 2 largely overspent
  first year budget
• Almost completely unfunded resources for test bed
  equipment, networking and security

8
Experience of first year and main issues

• The project is still up and running but,
• Needs quickly to implement EU recommendations
• Needs to understand how to progress with the test
  beds by mastering complexity, emerging standards
  and support issues
• Needs to adapt and connect to proliferation of
  Grid projects world-wide
• Needs to consolidate user communities (VOs)
• Needs to keep cost under control
• Needs to develop an after EDG strategy (LCG, FP6?)

9
First project review follow up

• Mobil agents explained why we dont need them
• Security features and cross-authentication
  explained our plans and EoI for a security
  project I FP6 submitted by a subset of partners
• Integration of new resources in the testbed
  explained WP6 approach and WP4 plans for a light
  weight installation procedure
• More flexible fabric installation procedures and
  requirements WP4 is following this
• Interface to Globus and participation towards a
  common evolution main issue under active
  discussion (also in the context of GGF, Glue etc.)

10
First project review follow up

• Scalability of resource scheduling architecture
  needs test-plan implemented and test group
  activated
• Improve data replica scheme being address for TB
  2
• Improve test use cases being discussed also in
  collaboration with DataTAG
• Involve more EO applications being addressed by
  WP9 also as part of their review
• Expand the use of Grid in EO being addressed by
  WP9 also as part of their review
• Improve project visibility and publish the
  content of EU deliverables being addressed by
  WP11 and WP12 (Information officer hired)

11
First project review follow up

• ESA review to address their role in the project
  and their expenditure profile carried out
  successfully last week in Frascati
• PO short-coming addressed by replacing second
  project admin.
• ESTEDI project contacted by WP2, publication of
  content of deliverables reviewed weekly
• GGF participation and GridStart
• Involvment of industrial partners to be better
  documented for next review
• EDG license agreed
• WP11 should improve quantitative measurement of
  dissemination, web content etc.

12
Grid aspects covered by EDG testbed 1
13
Testbeds

• Development
• Small set of sites (approx 5) with small clusters
  of PCs for development purposes to test new
  versions of the EDG middleware, configurations
  etc.
• Not stable constantly changing
• Currently testing EDG release 1.2
• Production
• More stable environment for use by application
  groups
• more sites
• more nodes per site (grow to meaningful size at
  major centres)
• more users per VO
• Currently running EDG release 1.1
• Hope to install EDG 1.2 during June

14
Software Release Procedure

• Coordination meeting
• Gather feedback on previous release
• Review plan for next release
• WP meeting
• Take basic plan and clarify effort/people/dependen
  cies
• Sw development
• Performed by WPs in dispersed institutes and run
  unit tests
• Software integration
• Performed by WP6 on frozen sw
• Integration tests run
• Acceptance tests
• Performed by Loose Cannons et al.
• Roll-out
• Present sw to application groups
• Deploy on testbed

themed tech. meets
testbed 1 Dec 11 2001 100 participants
Software release Plan http//edms.cern.ch/document
/333297
Roll-out. meeting
15
EDG platforms

• EDG 1.1 (now) supports Linux RedHat 6.2
• Globus2beta21
• EDG 1.4 (July) will support Linux RedHat 7.2
  6.2
• Globus2.x
• EDG 2.0 (October) will support Linux RedHat 7.2
  6.2
• Globus2.x
• In 2002 we will start port to Solaris as 3 steps
• WP1 UI tools
• Other middleware
• WP4 fabric mgmt tools

16
Release Plan details

• Current release EDG 1.1.4
• Deployed on testbed under RedHat 6.2
• Finalising testing of EDG 1.2
• WP1 APIs (see Massimos talk)
• WP2 GDMP 3.0 (see Heinzs talk)
• WP5 secure access to MSS (Castor)
• EDG 1.3 (end June)
• Build using autobuild tools to ease future
  porting
• INTERNAL RELEASE will not be deployed on
  production testbed

• EDG 1.4 (end July)
• Support RH 6.2 7.2
• Basic support for interactive jobs
• Use MDS 2.2 with first GLUE schema
• EDG 2.0 (Oct)
• Still based on Globus 2.x (pre-OGSA)
• Use updated GLUE schema
• Job partitioning check-pointing
• Advanced reservation/co-allocation
• Integration of Condor DAGman

See http//edms.cern.ch/document/333297 for
further details
17
John Gordons Testbed Trident
iVDGL Datagrid
LCG
Les Robertson, CERN

• Grid projects -
• responsible for testbeds on which developers
  can build and test their software
• testbeds for demos, Beta testing, scheduled
  data challenges

LCG - responsible for operating a production
Grid - 24 X 7 service - as easy to use
as LXBATCH
18
Issues when adding new sites to the testbed

• EDG is currently setting-up procedures explaining
  how to add new sites
• Variations already tested with Taiwan and Romania
• Step-by-step instructions produced which we
  expect to become simpler over time
• Need to clarify the minimum requirements for a
  site to become a member of the testbed
• A number of regular tasks have to be performed by
  the sites administrators
• A maximum delay needs to be defined for
  responding to requests/problems if the testbed is
  to run efficiently
• Sites from new countries have to identify/create
  a supporting CA
• Since CAs need mutual trust this could lead to an
  explosion of inspection activities
• Some tasks will fall on the people responsible
  for managing the VOs
• HEP experiment secretariats already perform some
  level of authentication of their institutes and
  members. How an we get some leverage from this?

19
GLUE
Antonia Ghiselli, INFN

• http//www.hicb.org/glue/GLUE-v0.04.doc
• Grid Laboratory Uniform Environment (GLUE) focus
  is on interoperability between Grid software
  projects.
• GLUE is a collaboration of grid projects with
  effort and management through iVDGL and DataTAG.
• GLUE will report to and get guidance from HICB
  and JTB.
• GLUE components are finalized to HEP experiments.
• To date participating experiments are Alice,
  Atlas, BaBAR, CDF, CMS, D0, (LHCb?)
• and projectsApGrid, CrossGrid,DataTAG, DataGrid,
  GriPhyN, IN2P3, INFN-Grid, iVDGL, LCG, GridPP,
  PPDG

20
Interaction with PPDG/GriPhyN/iVDGL

• Work with dataTAG via the InterGrid (GLUE) to
  investigate inter-operability of US and EU grids
• Authentication infrastructure
• perform cross organizational authentication
• DOE/EDG CAs mutual trust for Certs
• Unified service discovery and information
  infrastructure
• discover the existence and configuration of
  services offered by the testbeds
• Mature draft of information schema developed
• Data movement infrastructure
• move data from storage services operated by one
  organization to another
• Authorization services
• perform some level of cross organization,
  community based authorization
• Computational services
• coordinate computation across organizations to
  allow submission of jobs in EU to run on US sites
  and vice versa
• These steps have to matched with EDG software
  release plan for deployment

Done
Started
21
The LHC Computing Grid Project
Goal Prepare and deploy the LHC computing
environment

• applications - tools, frameworks, environment,
  persistency
• computing system ? services
• cluster ? automated fabric
• collaborating computer centres ? grid
• CERN-centric analysis ? global analysis
  environment
• foster collaboration, coherence of LHC
  regional computing centres
• central role of data challenges

This is not yet another grid technology project
it is a grid deployment project
Les Robertson, CERN
22
The futureOpen Grid Services Architecture

• Increasingly popular standards-based framework
  for accessing network applications
• W3C standardization Microsoft, IBM, Sun, others
• WSDL Web Services Description Language
• Interface Definition Language for Web services
• SOAP Simple Object Access Protocol
• XML-based RPC protocol common WSDL target
• WS-Inspection
• Conventions for locating service descriptions
• UDDI Universal Desc., Discovery, Integration
• Directory for Web services

• Service orientation to virtualize resources
• Everything is a service
• From Web services
• Standard interface definition mechanisms
  multiple protocol bindings, local/remote
  transparency
• From Grids
• Service semantics, reliability and security
  models
• Lifecycle management, discovery, other services
• Multiple hosting environments
• C, J2EE, .NET,

Globus 3.0 (OGSA compliant) end 2002 Industry
involvement IBM, MicroSoft, Sun
23
Future Plans

• Collaborate with closely related activities
  (Glues, iVDGL, EDT, X) to deliver the basis of a
  production testbed to LCG
• Extend collaboration with US based LHC activity
  via EDT/iVDGL
• Prepare for second test bed in autumn 2002
• Promote early standards adoption with
  participation to GGF and other international
  bodies

24
Conclusion

• Is there life after EDG?
• EoI submitted for a large III (Integrated
  Infrastructure Initiative) in coordination with
  European RNs (RN Geant follow on)
• 300 M Euros total budget, 100 M Euros EU funding
• Overwhelming interest from European Grid
  community and industry
• Will our community be able to deliver up to these
  very high expectations?
• What is LCG role into this? and the other HEP
  national Grid activities?
• Interesting discussions today and in the year
  ahead

25
Learn more about Grids DataGrid
Programme includes Grid Lectures by Ian
Foster Carl Kesselman Hands-on tutorial DataGrid
Apply now via web http//csc.web.cern.ch/CSC/ Plac
es are limited
CERN School of Computing 2002
Vico Equense, Italy, 15-28 September 2002 The
2002 CERN School of Computing is organised by
CERN, with the Institute of Composite and
Biomedical Materials, National Research Council,
Naples, Italy.