Workshop on Sustainable National Grid Services

1
Workshop on Sustainable National Grid
Services Edinburgh, Feb 22 23, 2007 D-Grid
Progress Towards Sustainability Wolfgang
Gentzsch D-Grid
2

• Todays Topics
• A little history of D-Grid
• D-Grid a few details
• 2 ways towards a sustainability strategy
• 1. Learn from others (analysis of major
  grids)
• 2. Learn from our own requirements
• Analyzing major grid projects
• D-Grid Sustainability Workshop
• Conclusions

3
History of D-Grid Initiative

• 01/2003 German scientists started D-Grid
  Initiative (UK pressure)
• report with recommendations
  for German Government
• 03/2004 BMBF announced 100 ME e-Science
  Initiative for Germany
• 08/2004 BMBF Call for Proposals for e-Learning,
  Knowledge
• Networks, and Grid
  Computing
• 09/2005 D-Grid-1 25 ME, early adopters,
  Services for Science
• 06/2007 D-Grid-2 new communities and services
  providers
• 06/2008 D-Grid-3 (?) Service Grids for
  research, industry, society

4

• Learn from others, collaborate with others
• Sustainability in architecture (standards),
  technology (robust),
• users (applications), market, legal,
  government,
• Start with a plan for sustainability
• Users and applications drive sustainability
  (not only!)
• Develop clear benefits for users
• Make everything easy to use
• Political and policy landscape has to be right

Sustainability, Ideas in the Beginning
5
D-Grid
D-Grid-1
Knowledge Management
Astro-Grid
C3-Grid
HEP-Grid
IN-Grid
MediGrid
ONTOVERSE
WIKINGER
WISENT
Textgrid
Im Wissensnetz
. . .
Generic Grid Middleware and Grid Services

• Integration Project

Courtesy Helmut Loewe, BMBF
6
D-Grid-2
D-Grid-1 2
Knowledge Management
Astro-Grid
C3-Grid
HEP-Grid
IN-Grid
MediGrid
ONTOVERSE
WIKINGER
WISENT
Textgrid
Im Wissensnetz
. . . . . .
Services Level Agreements
Generic Grid Middleware and Grid Services

• Integration Project

7
D-Grid-3
D-Grid-1 2 3
Knowledge Management
Astro-Grid
C3-Grid
HEP-Grid
IN-Grid
MediGrid
ONTOVERSE
WIKINGER
WISENT
Textgrid
Im Wissensnetz
. . . . . .
Knowledge Management
Services Level Agreements
Generic Grid Middleware and Grid Services

• Integration Project

8
D-Grid-3
D-Grid-1 2 3
Knowledge Management
Astro-Grid
C3-Grid
HEP-Grid
IN-Grid
MediGrid
ONTOVERSE
WIKINGER
WISENT
Textgrid
Im Wissensnetz
. . . . . .
Virtual Competence Centers for Middleware,
Resources, Support, Knowledge
Services Level Agreements
Generic Grid Middleware and Grid Services
9
D-Grid Middleware Stack
User
Application Development and User Access
GAT API
Plug-In
GridSphere
UNICORE
Nutzer
High-levelGrid Services
SchedulingWorkflow Management
Monitoring
LCG/gLite
Data management
Basic Grid Services
AccountingBilling User/VO-Mngt
Globus 4.0.1
Security
Resourcesin D-Grid
DistributedCompute Resources
NetworkInfrastructur
DistributedData Archive
Data/Software
10
Learn from Others
11
e-Science Grid Initiatives Investigated
Initiative Time Funding People )
Users UK e-Science-I 2001 - 2004 180M
900 Res. UK e-Science-II 2004 - 2006 220M
1100 Res. Ind. TeraGrid-I 2001 - 2004
90M 500 Res. TeraGrid-II 2005 -
2010 150M 850 Res. ChinaGrid-I 2003 -
2006 20M RMB 400 Res. ChinaGrid-II 2007
2010 50M RMB ) 1000 Res. NAREGI-I 2003 -
2005 25M 150 Res. NAREGI-II 2006 -
2010 40M ) 250 Res. Ind. EGEE-I
2004 - 2006 40M 800 Res. EGEE-II 2006
- 2008 45M 1000 Res. Ind. D-Grid-I 2005
- 2008 25M 220 Res. D-Grid-II 2007 -
2009 25M 220 ( 440) Res. Ind. )
estimate
12
Main Objectives of e-Science Projects
UK e-Science To enable the next generation of
multi-disciplinary collaborative science and
engineering, to enable faster, better or
different research. EGEE To provide a seamless
Grid infrastructure for e-Science that is
available for scientists 24 hours-a-day. ChinaGri
d To provide a research and education platform
by using grid technology for the faculties and
students among the major universities in
China. NAREGI To do research, development and
deployment of science grid middleware. TeraGrid C
reate a unified Cyberinfrastructure supporting a
broad array of US science activities using the
suite of NSF HPC facilities D-Grid Build and
operate a sustainable grid service infrastructure
for German research (D-Grid1) and research and
industry (D-Grid2)
13
Components of e-Infrastructures for Science(Tony
Hey, 2003)

1. Resources Networks with computing and data
   nodes, etc.
2. Development/support of standard middleware grid
   services
3. Internationally agreed AAA infrastructure
4. Discovery services and collaborative tools
5. Data provenance, curation and preservation
6. Open access to data and publications via
   interoperable repositories
7. Remote access to large-scale facilities
   Telescopes, LHC, ITER, ..
8. Industrial collaboration

Ideally having well-defined specific service
support centres Examples, UK
OMII, DCC, NGS
14
Grid Middleware Stacks, major modules
UK e-Science Phase 1 Globus 2.4.3, Condor,
SRB. Phase 2 Globus 3.9.5 und 4.0.1, OGSA-DAI,
Web services. EGEE gLite distribution
elements of Condor, Globus 2.4.3 (via VDT
distribution). ChinaGrid ChinaGrid
Supporting Platform (CGSP) 1.0 is based on Globus
3.9.1, and CGSP 2.0 is implemented based on
Globus 4.0. NAREGI NAREGI middleware and Globus
4.0.1 GSI and WS-GRAM TeraGrid GT 2.4. and
4.0.1 Globus GRAM, MDS for information, GridFTP
TGCP file transfer, RLS for data replication
support, MyProxy for credential mgmnt D-Grid
Globus 2.4.3 (gLite) and 4.0.3, Unicore 5,
dCache, SRB/iRODS, OGSA-DAI, GridSphere, GAT,
VOMS and Shibboleth
15
Sustainability UK
e-Science National Grid Service (NGS), Grid
Operations Support Center (GOSC), National
e-Science Center (NeSC), Regional e-Science
Centers, Open Middleware Infrastructure
Institute (OMII), Digital Curation Center
(DCC) EGEE Plans to establish a European Grid
Initiative (EGI), together with NGIs, to provide
persistent grid service federating national grid
programmes starting in 2008 ChinaGrid
Increasing numbers of grid applications using
CGSP grid middleware packages NAREGI Software
will be managed and maintained by Cyber Science
Infrastructure Center of National Institute of
Informatics TeraGrid NSF Cyberinfrastructure
Office 5 year Coop. Agreement. Partnerships with
peer grid efforts and commercial web services
activities in order to integrate broadly
D-Grid DGI WP 4 sustainability, services
strategies, and business models
16
e-Science Applications drive Sustainability
UK e-Science Particle physics, astronomy,
chemistry, bioinformatics, healthcare,
engineering, environment, pharmaceutical,
petro-chemical, media and financial sectors
EGEE 2 pilot applications (physics, life
science) and applications from other 7
disciplines. ChinaGrid Bioinformatics, image
processing, computational fluid dynamics, remote
education, and massive data processing NAREGI N
ano-science applications TeraGrid Physics
(Lattice QCD calculations, Turbulence
simulations, Stellar models), Molecular
Bioscience (molecular dynamics), Chemistry,
Atmospheric Sciences D-Grid-1 Astrophysics,
high-energy physics, earth science, medicine,
engineering, humanities
17

• D-Grid Towards a Sustainable
• Infrastructure for Science and Industry
• Govt is changing policies for resource
  acquisition (HBFG ! )
• to enable a service model
• 2nd Call Focus on Service Provisioning for
  Sciences Industry
• Strong collaboration with Globus Project,
  EGEE, Deisa,
• CrossGrid, CoreGrid, GridCoord, GRIP,
  UniGrids, NextGrid,
• Application and user-driven, not
  infrastructure-driven
• Focus on implementation and production, not
  grid research, in a
• multi-technology environment (Globus,
  Unicore, gLite, etc)
• D-Grid is the Core of the German e-Science
  Initiative

18
Challenges for Research and Industry

• Sensitive data, sensitive applications (medical
  patient records)
• Different organizations get different benefits
• Accounting, who pays for what (sharing!)
• Security policies consistent and enforced across
  the grid !
• Lack of standards prevent interoperability of
  components
• Current IT culture is not predisposed to sharing
  resources
• Not all applications are grid-ready or
  grid-enabled
• Open source is not equal open source (read the
  small print)
• SLAs based on open source (liability?)
• Static licensing model dont embrace grid
• Protection of intellectual property
• Legal issues (e.g. FDA, HIPAA, multi-country
  grids)

19
Lessons Learned and Recommendations

• During development, operation, the grid
  infrastructure should be modified and improved in
  large cycles only all applications depend on
  this infrastructure !
• Continuity especially for the infrastructure part
  of grid projects is important. Therefore, funding
  should be available after the project, to
  guarantee services, support and continuous
  improvement and adjustment to new developments.
• Interoperability Use software components and
  standards from open-source and standards
  initiatives especially in the infrastructure and
  application middleware layer.
• Close collaboration is mandatory between
  developers of the grid infrastructure and the
  applications to best utilize grid services and to
  avoid application silos.
• Infrastructure should be user-friendly for easy
  adoption for new communities. The infrastructure
  group should offer installation/operation service
  and support.

20
Lessons Learned and Recommendations

• For complex projects (infrastructure and
  application projects), a management board
  (consisting of the leaders of the different
  projects) should steer coordination and
  collaboration among the projects.
• On top of grid infrastructure, new projects
  should utilize the generic infrastructure and
  focus on an application or on a specific service,
  to avoid complexity and re-inventing wheels and
  building grid application silos. .
• Centers of Excellence should specialize on
  specific services, e.g. integration of new
  communities, grid operation, utility services,
  training, support, etc.
• Participation of industry has to be
  industry-driven. Push from outside, even with
  government funding, is not promising. Success
  will come only from real needs e.g. through
  existing collaborations with research and
  industry, as a first step.
• Implement utility computing in small steps,
  enhancing existing service models moderately,
  testing utility models first as pilots. Often,
  todays government funding models are
  counter-productive for utility services.

21

• Workshop
• Sustainability in D-Grid
• Oct 9 10 2006
• Sustainability in Grids
• S. and the funding Organization (Govt)
• S. and monitoring, accounting, billing
• S. of the D-Grid Infrastructure
• S. and application communities
• Example DFN German Research Network
• S. and Industry
• S. and support
• The European Grid Initiative (EGI)

22

• Results of the Workshop
• Requirements
• There is a general need for a sustainable
  infrastructure
• Funding agency demands cost-neutral operation
• But not only monetary considerations, but also
  research
• Benefits for all constituencies
• Long-term data preservation
• International integration
• Acceptance of infrastructure through ease of use
• Long-term planning safety for grid communities

23

• Results of the Workshop
• Challenges
• Heterogeneous middleware complicates building
  sustainable grid
• Today user unfriendly and complex environments
• Integration of new hardware from new partners
  and communities
• Currently, D-Grid is not a legal entity
• Long-term financing of resources and their usage
  is not clear
• Grid-enabled software licensing model is unclear
• Broadening community grids beyond their current
  core members
• Germany Laender investments restricted to
  local usage

24

• Results of the Workshop
• Steps towards a sustainable infrastructure
• Increase significantly the number of grid users
• Govt funding for D-Grid specific resources was
  key
• Support of several middlewares important
• Long-term goal independence of D-Grid from
  funding
• Encourage Govt to change current funding
  policies for resources
• User-friendly user support of utmost importance
  (DGI CGs)
• Industry participation as users (SMEs) and
  providers (IT companies)

25

• Results of the Workshop
• Conclusions and Recommendations
• D-Grid seems to be on track towards a
  sustainable infrastructure
• A centralized resource infrastructure is
  important, but the how
• still has to be discussed (DGI vs CGs)
• Implementation of sustainable D-Grid only
  together with users (CGs)
• Sustainable usage (business) models only with
  users (CGs)
• Integration of D-Grid in European infrastructure
  is important
• Central D-Grid institution should encourage
  broad acceptance
• of D-Grid, incl certification of and support
  for resources
• Role of industry unclear, but participation
  possible today

26
Lessons Learned and Recommendations

• Continuity Grid infrastructure should be
  modified and improved in large cycles only
  applications depend on infrastructure !
• Sustainability Funding should be available
  after end of project, to guarantee services,
  support and continuous improvement.
• Interoperability Use open-source software and
  standards especially in the infrastructure and
  application middleware layer.
• Collaboration between infrastructure developers
  and the applications, to best utilize grid
  services and to avoid application silos.
• User-Friendliness for easy adoption for new
  communities. Infrastructure group should offer
  installation, operation and support services.
• Grid Services Centers of Excellence should
  specialize on specific services, e.g. integration
  of new communities, grid operation, utility
  services, training, support, etc.
• Participation of Industry has to be
  industry-driven. Push from outside, even with
  govmnt funding, is not promising. Success comes
  only from real needs e.g. through already
  existing collaborations between research and
  industry.

27
Many Thanks to

• UK-e-Science Tony Hey, Steven Newhouse, Carole
  Goble, Malcolm Atkinson, John Darlington, Trevor
  Cooper Chadwick, Monica Schraefel, Luc Moreau,
  Paul Watson, Aaron Turner
• TeraGrid Charlie Catlett, Dane Skow
• ChinaGrid Hai Jin
• Naregi Kazushige Saga, Satoshi Matsuoka,
  Kenichi Miura
• EGEE Bob Jones, Dieter Kranzlmueller, Erwin
  Laure
• D-Grid Uwe Schwiegelshohn, Wolfgang Guerich,
  Klaus Ullmann, Klaus Peter Mickel, Matthias
  Steinmetz, Matthias Kasemann, Wolfgang Hiller,
  Otto Rienhoff, Michael Resch, Elmar Mittler,
  Wilhelm Hasselbring
• RENCI Dan Reed and Alan Blatecky