GT2 and GT3 experiences on INWA and HPC Europa

1
GT2 and GT3 experiences on INWA and HPC Europa

• UK Globus Week 7th April 2005

2
Overview

• INWA - use of GT2 and GT3
• http//www.epcc.ed.ac.uk/inwa
• HPC Europa use of GT 3
• http//www.hpc-europa.org/
• GT 4 wish-list

3

• INWA

4
INWA Background

• Funded by UK Economic Social Research Council
  (UK) in the Pilot Projects in E-Social Science
• Small scale projects to explore the potential of
  Grid technologies within the social sciences
• Informing Business Regional Policy Grid
  enabled fusion of global data local knowledge
• INWA Innovation Node Western Australia
• Project Aims
• Evaluate the suitability of existing grid
  solutions for secure distributed data mining and
  analysis on commercially sensitive data
• Investigate the advantages of fusing public and
  private data enabled by a grid environment
• Two funding phases
• 1st phase November 2003 to August 2004
• set up INWA grid between UK and Australia
• data mining between sites over the grid
• 2nd phase started November 2004, due to finish
  May 2005
• Addition of a node in China to the existing INWA
  grid
• Basic data mining over the grid

5
Grid software employed

• Transfer-queue Over Globus (TOG) v1.1 from the UK
  e-Science Sun Data and Compute Grids project
• provides access to remote HPC resource
• Uses Globus Toolkit 2.4
• Open Grid Services Architecture Data Access and
  Integration (OGSA-DAI) Release 3.1
• provides access control and discovery of
  distributed heterogeneous data resources
• Uses Globus Toolkit 3.0
• First Data Investigation on the Grid (FirstDIG)
• grid data service browser provides SQL access to
  OGSA-DAI enabled resources
• now part of OGSA-DAI R4.0/5.0
• Globus Toolkit 2.4 and 3.0
• Grid middleware

6
The INWA Grid
7
TOG (Transfer-queue Over Globus)
Site B
Site A
Grid Engine
User B
Grid Engine
User A
e
g
h
f
a
b
c
d
Globus 2.2.x
d
e
Transfer queue

• Integrates Grid Engine and Globus 2.2.x/2.4
• Globus GSI for security, GRAM for interaction
  with remote GE
• GASS for small data transfer, GridFTP for large
  datasets
• Written in Java - Globus functionality accessed
  through Java COG kit

8
TOG/GridEngine/Globus set-up
9
UK- Australia lessons learned

• Performing Data Integration
• TimeZone date problems
• Dates are stored as a time so
• 600am Dec 25th in Perth Australia is converted
  to
• 1000pm Dec 24th in Edinburgh, UK
• If data is processed in the UK, the wrong date is
  used.
• Security issues
• Bugs in
• Globus JavaCoG in GT3
• OGSA-DAI could not switch security for Grid data
  transfers
• TOG had no security option
• All of these have been fixed
• Middleware not mature enough for commercial
  deployment
• Not out-of-the box significant effort to build
  grid
• Bug fixes were required
• Sys admin skills still necessary to maintain the
  grid
• Scalability- difficulty with large results in
  OGSA-DAI V3.1
• Fixed in OGSA-DAI V4.0

10
UK-Australia-China lessons learned

• Reverse Domain Name Service (DNS lookup)

tms_at_e3500 nslookup -sil 129.215.56.231
Server 129.215.56.230 Address
129.215.56.23053 231.56.215.129.in-addr.arpa
name e3500.epcc.ed.ac.uk

• Required by GT 2 only for sustaining
  connections not establishing
• But
• In China, few IP addresses relative to demand
• Usually not possible to configure reverse DNS
  look-up at same DNS server that handles usual
  forward DNS lookup
• Had to explicitly configure INWA participating
  machines

11

• HPC Europa

12
HPC Europa

• Full title Pan-European Research infrastructure
  on High Performance Computing for the Science of
  the 21st Century
• Goal to provide advanced computational services
  in an integrated way to the European Research
  community
• 14 partners across Europe
• Project activities
• Transnational Access Programme
• Networking Activities
• Joint Research Activities (JRA1, JRA2)

13
JRA2 Single Point of Access

• Motivation
• To provide a uniform access to resources of all
  centres, transparently and regardless of physical
  location
• To achieve this
• Building a HPC Europa portal to provide access
• Develop and adapting the necessary tools
• Participating JRA2 centres have existing
  middleware brokers on top of either Globus and
  Unicore
• Developing a generic portal to sit on top of
  these
• For EPCC this means
• JOSH (JOb Scheduling Hierachically) with GT3.2
  and Grid Engine

14
JOSH (JOb Scheduling Hierarchically)

• Based on Globus 3 and grid services
• Adds a new 'hierarchical' scheduler above Grid
  Engine
• Command line interface
• hiersched submit_ge
• Takes GE job script as input (embellished with
  data requirements)
• Queries grid services at each compute site to
  find best match and submits job
• Job controlled through resulting 'job locator

15
GT 3.2 Issues

• MMJFS
• Rogue UHE processes blocking ports
• Ease of installation
• Still requires significant skills/time
• Documentation
• NOAA experimental Grid between FSL and PMEL
  using JOSH, GT3.2, Grid Engine
• Did not use for a few days, came back, it had
  stopped working, found error message in GT pages,
  no fix for it
• A very frustrated Globus user
• Reinstalled GT3.2 and it all worked again but
  does not know why

16

• GT 4 wish-list

17
GT 4 wish-list

• Easier installation and maintenance
• Migration path from GT 3.2 to GT 4
• Documentation