Grid Interoperability Update on GridFTP tests

1
Grid InteroperabilityUpdate on GridFTP tests

• Gregor von Laszewski
• gregor_at_mcs.anl.gov
• http//www.cogkit.org

2
Outline

• Proposal Documentation consolidation
• Proposal GIN GridFTP Report
• Proposal for Survey
• Proposal for User Identification
• Proposal for identification of what tests to
  execute
• Update GridFTP
• Java CoG Kit Workflow (Testing) Framework
• Testing GridFTP on the TeraGrid
• Testing GridFTP on the GIN Resources
• Summary

3
Proposal Documentation Making Interoperability
possible

• One aspect of interoperability is based on an
  effective mechanism to document and distribute
  the information.
• Proposal Consolidate Documentation
• Today information is spread across several web
  sites and Wikis
• Build a single entry point for the working group
  documentation
• Proposal Improve the contents
• Work on extended report, publishable article
• Most important from my point of view
• Improve the documentation and description on how
  to get accounts on the other Grids
• Proposal Test documentation effort
• Have interoperability tests executed by third
  parties to test effectiveness of documentation
  and approaches
• Engage in an iterative improvement process of the
  documentation

4
GIN Today Documentation

• Documentation links (We need your help to add to
  this page)
• This should be done during the workshop or as
  response to the mail
• http//wiki.nesc.ac.uk/read/gin-jobs?HomePage
• https//forge.gridforum.org/sf/wiki/do/viewPage/pr
  ojects.gin/wiki/GINData
• Problem with OGF/GGF infrastructure
• Gridforge is not used (is it too complicated? Is
  it too much overhead)
• Only charter is published
• Wikis are the tool of choice for the working
  group as much of the information may change
  quickly. Proposal use a single Wiki
• If wiki is used a feature is needed to archive
  the Wiki contents in a single document.
• OGF schedules do not have pointers to the working
  group
• Google search in OGF Web page is ineffective
• searching for GIN must give as first link the
  working group page
• Search in Gridforge is non operational
• Example search for GIN in wikis on Gridforge
  returns no results
• Links in the mailing list archives are no longer
  operational
• Example links to documentation that was send by
  Erwin in May 06
• Transition to OGF has not been completed.

5
Proposal GIN GridFTP Report

• Goal provide an extended document that includes
  a formal way of reporting on the activities of
  this particular subtask as part of the GIN
  working group in a single report.
• Assure that the goals overlap with the working
  group goals
• Proposed Features that the report may address (in
  the area of GridFTP)
• Survey of different implementations of GridFTP
  servers and clients
• Identification of the users for interoperability
  tests and their goals
• Administrators
• Developers
• Users
• Survey of different tools that monitor GridFTP
  behavior in production Grids
• Identification of which tests need to be
  performed to satisfy
• Administrators
• Developers
• Users
• Links to programs and documentation that can
  perform the tests.

6
Administrators - Developers - Users

• There are different dimensions to be considered
  when addressing interoperability issues
• While administrators may care about setting up
  tests between virtual organizations, individual
  user needs are not completely addressed by this
  test as they are not conducted by the user
  itself.
• Lessons learned over the past 5 years have shown
  that users MUST have the ability to perform a set
  of interoperability test themselves. This
  includes in most case the answer to the
  questions
• Can I copy a file form here to there?
• What is the performance I get when I do the
  default?
• Administrator may be more interested to obtain
  the performance and throughput of the service.
• Test conducted for each individual user are
  typically not conducted.
• Developers may be more concerned about functional
  or protocol interoperability.
• GIN must address not only administrators and
  developers, but more importantly the users.

7
Update GridFTP Testing with the Java CoG Kit
Workflow

• Java CoG Kit Workflow
• Java CoG Kit workflow can be run through a remote
  service, an API, or a command line tool.
• Java CoG Kit contains easy to use workflow
  language that in the past has been able to bridge
  between different Globus middleware (GT2, GT3,
  GT4, SSH, )
• Language features include
• Functional programming language expressible in
  XML like syntax
• Variable Futures
• Explicit parallelism
• HTML form generation (XUL like)

• Two concepts
• Abstractions copy a file from here to there
  regardless of the implementation / protocol
• Providers Mapping into Grid middleware or
  commodity technologies

8
cont . Java CoG Kit workflow

• Transfers are real simple, specify source,
  destination, and provider

• ltprojectgt
• ltinclude file"cogkit.xml"/gt
• ltexecute executable"/bin/date"
• stdout"thedate"
• host"hot.mcs.anl.gov"
  provider"GT2"/gt
• ltecho message"Job completed. Transferring the
  output"/gt
• lttransfer srchost"hot.mcs.anl.gov"
  srcfile"thedate"
• desthost"localhost"
  provider"gridftp"/gt
• ltecho message"Transfer complete"/gt
• ltset name"date"gt
• ltreadFile file"thedate"/gt
• lt/setgt
• ltecho message"The date is date"/gt
• lt/projectgt

• Technically we could add other providers such as
  nordu-GridFTP,RFT, SRM,

• It is obvious that this can be used to test not
  only the file transfers but also job submissions
  and other things.
• See Also
• http//wiki.cogkit.org/index.php/Java_CoG_Kit_Docu
  mentation

9
What could be tested

• File transfer
• Directory creation
• Permissions
• 2nd party transfers
• 3rd party transfers
• All the features that are available through the
  Java Cog Kit abstractions
• We can integrate more features if necessary
• Job execution
• Similar
• Other things

• The CoG workflow and its embedded HTML
  capabilities make it possible to create custom
  tests.

10
What did we test?

• First we wanted a simple test
• Transfer a file form here to there
• Perform the test by someone outside of the CoG
  Kit group to identify if the tool is
• Easy to install
• Functional
• Robust
• Compare if the tests are similar to the once
  obtained from Globus native GridFTP clients

• Result
• In all aspects successful
• Through a single shell script the testing
  framework can be installed and run
• The results have been confirmed with Raj, who is
  a member of the GridFTP development team.
• To Do
• We moved to an SVN archive this month and need to
  update that information so others can replicate
  it if they wish)

11
GridFTPTest.sh

• Installing and Running
• 1. Requirements Apache Ant and a Java SDK
• 2. Mkdir gin
• 3. Get cog-ftptest-install.sh script from Java
  CoG Kit SVN and place in gin
• 4. Run cog-ftptest-install.sh
• 5. Edit the hosts file and customize the set of
  sites to include all hosts you like to test
• host("ltnamegt
• service(provider"gsiftp", type"file",
  url"gsiftp//lthostgtltportgt")
• )
• 6. Run the tests
• cog-workflow runtests.k
• 7. The output files should appear in the
  "output" directory
• Note We just moved to an SVN directory and some
  issue with sourceforge SVN causes a problem upon
  download. Sourcforge is working with us to
  resolve it.
• In future we intend to host this in dev.globus.org

12
GridFTP tests on on TeraGrid (Conducted by Raj
K.)

• Verification
• Download
• Install
• Configure
• Run
• No problems
• Tool worksas promised
• Raj is a GridFTP developer and confirmed he gets
  the same results with the GridFTP tools

13
GridFTP test on GIN resources (Conducted by Raj
K.)

• Just changing the hosts to the GIN resources
  results in
• Most Issues are related to the user not being
  able to authenticate/authorize

14
Next Steps

• Complete the documentation of the tool
• Complete the SVN transition
• Have a user apply to all GIN Grid resources and
  make sure we do not have authentication issues.
• I thought this is trivial, but in fact it is not.
• Even on TG this is still a challenge today as it
  is not as simple as just applying for an account.

15
Summary GIN gridFTP from a users perspective

• It is possible to use existing tools such as the
  CoG Kit to write interoperability tests that can
  be executed by the user.
• Tools such as the Java CoG Kit even help briding
  interoperability issues
• The issue is at this time not interoperability of
  gridftp servers, we can simply use the
  appropriate client. The Java CoG kit has
  demonstrated that multi-client abstractions are a
  feasible approach. Hence through a higher level
  of abstraction we can provide a considerable
  notion of interoperability.
• Surprisingly we found that the todays real issue
  is that a user has to spend days/weeks to get
  onto the Grid due to post account bootstraping of
  the users environment to use the Grid.
• Proposal We should as community make it possible
  that users can get on the Grid fast
• If it takes a user more than one day to get
  access to the about 30 Grid resources we tested
  with this effort need to do something about it.

16

• Testing GridFTP on the TeraGrid
• Testing GridFTP on the GIN Resources

17
Summary