6a'1

1
Globus ToolkitI - Basic structure, and job
submission
2
Globus

• Open source grid software toolkit that has been
  developing since late 1990s. Provides
• An Underlying Grid Security Infrastructure (GSI)
• Resource Management
• Data Management
• Information services.
• Higher level tools are meant to be implemented
  above these basic services

3
Grid Security Infrastructure

• Provides security functions including
• Authentication
• Authorization
• Delegation
• Confidential Communication

4
Resource Management

• Job submission
• Job status
• Basic resource allocation
• Globus does not have its own job scheduler to
  find resources and automatically send jobs to
  suitable machines. For that use a separate
  scheduler - we will use Condor-G.

5
Information Services

• Support for gathering information and querying
  information.
• Based upon Lightweight Directory Access Protocol
  (LDAP)

6
Data Management

• Support for transferring files between machines

7
Key Components

• GSI (Grid Security Infrastructure)
• Grid security.
• GRAM (Grid Resource Allocation Manager)
• Remote job submission and control.
• GridFTP
• Secure data transfer.
• MDS (Monitoring and discovery Service)
• Interface to system and service information.

8

• Version 2
• (pre-2004)
• Contained named components
• GSI
• GRAM
• GridFTP
• MDS.
• Relatively stable.

9
From Introduction to Grid Computing with
Globus, IBM Redbooks, SG24-6895-012003, Fig. 4-6.
10
Globus version 2
From Introduction to Grid Computing with
Globus, IBM Redbooks, SG24-6895-012003, Fig. 7-2.
11
Globus version 2
From Introduction to Grid Computing with
Globus, IBM Redbooks, SG24-6895-012003, Fig. 7-3.
12
Resource Specification LanguageRSL

• Provides a specification for
• Resource requirements - machine type, number of
  nodes, memory, etc.
• Job description - directory, executable,
  arguments, environment

13

• Version 1 (Pre GT3) - a metalanguage describing
  job and its required execution. Specification
  given in an rsl file
• Version 2 (GT3) - specification described in a
  XML document with a schema.

14
RSL Version 1Constraints ExampleConjunction
(AND)

• To create 3-5 instances of myProg, each on a
  machine with at least 64 Mbytes memory available
  to me for 1 hours
• (executablemyProg)
• (countgt3)(countlt5)(memorygt64)
• (max_time60)

15
RSL Version 1Constraints ExampleDisjunction
(OR)

• To create 5 instances of myProg, each on a
  machine with at least 64 Mbytes memory or 7
  instances of myProg, each on a machine with at
  least 32 Mbytes memory
• (executablemyProg)
• (((count5)(memorygt64))
• ((count7)(memorygt32)))

16
RSL version 1Requesting multiple
resourcesmultirequest

• To execute 5 instances of myProg1 on a machine
  with at least 64 Mbytes memory and execute 2
  instances of myProg2
• ((count5)(memorygt64))
• (executablemyProg1))
• ((count2)(executablemyProg2))

17

• Can specify different resource managers on
  different machines using resourceManagerContact
  attribute

18
Can specify different resource managers on
different machines using resourceManagerContact
attribute.

• Example
• ( (resourceManagerContact
• flash.isi.edu754/CUS//CNflash.isi.edu-fork)
  
• (count1)
• (executablemy_appl1)
• )
• ( (resourceManagerContact
• sp139.sdsc.edu8711/CUS//CNsp097.sdsc.edu-lst
  )
• (count2)
• (executablemy_appl2)
• )

19
RSL creation with Globus version 2

• GT2 globus-job-run can be used to generate RSL
  from command line arguments with -dumprsl flag
• -help gives options

20

• We used Globus version 2.4 in a Supercomputing
  2003 demo organized by the University of
  Melbourne.
• 21 countries involved, numerous sites.

21
(No Transcript)
22
(No Transcript)
23
Version 3

• A re-implementation based upon the Open Grid
  Service Architecture (OGSA).
• OGSI-compliant Java interfaces.
• Contains GSI, GRAM, GridFTP, and MDS.
• Includes additional Web service components, some
  built on top of OGSI.
• Version 3.2 (March 2004) a major revision of
  version 3.0. Even commands changed.

24
From http//www.globus.org
25
GT 3 structureEarly representation
Non-GT3 services
Replica management for large data sets, ...
Job management, etc.
SSL, Certificates
OGSI (Grid services)
26
GT3More recent representation
27
Grid Service Container

• Contains all the files of GT3 system shielding
  it from environment.
• White part is the GT3 core
• Operates with a web service engine - stand-alone
  web service container provided for test purposes.
• Tomcat can be used.

28
Globus 3.2

• Has suite of command line tools.
• To start container
• globus-start-container
• (from GLOBUS-LOCATION directory). Should see a
  list of available services including any you have
  created.

29
GUI

• GT3 also has a Java based GUI service browser.
  To start
• globus-service-browser
• (from GLOBUS-LOCATION directory).

30
Double click service to invoke it

• From Globus Toolkit 3.0 Quick Start, IBM, Sept
  2003, http//www.redbooks.ibm.com/redpapers/pdfs/r
  edp3697.pdf For educational purposes only.

31

• From Globus Toolkit 3.0 Quick Start, IBM, Sept
  2003, http//www.redbooks.ibm.com/redpapers/pdfs/r
  edp3697.pdf For educational purposes only.

32
Resource Management issues in a Grid

• Sites owned by others
• Different Systems and Software
• Different Policies, ...

33
GT 3.2 GRAMGlobus Resource Allocation Manager

• A set of OGSI compliant services provided to
  start remote jobs. notably
• Master Managed Job Factory Service (MMJFS).
• Also a set of non-OGSI compliant services
  (Gatekeeper, Jobmanager) from pre-GT3.

34
(No Transcript)
35
Resource Specification Language Version 2 (RSL
-2)

• Now an XML schema.
• Requirements specified in RSL-2 schema in an XML
  file.
• Can specify everything from executable, paths,
  arguments, input/output, error file, number of
  processes, max/min execution time, max/min
  memory, job type etc. etc.

36
RSL-2

• Much more elegant and flexible, and in keeping
  with systems using XML.
• Can use XML parsers.
• Allows more powerful mechanisms with job
  schedulers.
• Resource scheduler/broker applies specification
  to local resources.

37
RSL-2 ExampleSpecifying Executable(executable/b
in/echo)

• ltgramexecutablegt
• ltrslpathgt
• ltrslstringElement value"/bin/echo"/gt
• lt/rslpathgt
• lt/gramexecutablegt

38
RSL-2 ExampleSpecifying Directory(directory/bi
n)

• ltgramdirectorygt
• ltrslpathgt
• ltrslstringElement value"/bin/"/gt
• lt/rslpathgt
• lt/gramdirectorygt

39
RSL-2 ExampleSpecifying Number(count1)

• ltgramcountgt
• ltrslinteger value"1"/gt
• lt/gramcountgt

40
RSL-2 ExampleSpecifying Arguments(argumentsHel
lo World)

• ltgramexecutablegt
• ltrslpathgt
• ltrslstringElement value"/bin/echo"/gt
• lt/rslpathgt
• lt/gramexecutablegt

41
RSL-2 ExampleSpecifying input, output, and
errorstdin/dev/null, stdout"stdout,
stderr"stderr"

• ltgramstdingt
• ltrslpathgtltrslstringElement value"/dev/null"/gt
  lt/rslpathgt
• lt/gramstdingt
• ltgramstdoutgt
• ltrslpathArraygt
• ltrslpathgt
• ltrslsubstitutionRef name"HOME"/gt
• ltrslstringElement value"/stdout"/gt
• lt/rslpathgt
• lt/rslpathArraygt
• lt/gramstdoutgt
• ltgramstderrgt
• ltrslpathArraygt
• ltrslpathgt
• ltrslsubstitutionRef name"HOME"/gt
• ltrslstringElement value"/stderr"/gt
• lt/rslpathgt
• lt/rslpathArraygt
• lt/gramstderrgt

42
RSL and RSL-2 Comparison for program echo(echo
used in assignment 3)

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltrslrsl
  xmlnsrsl"http//www.globus.org/namespaces/2003/0
  4/rsl"
• 
  xmlnsgram"http//www.globus.org/namespaces/2003/
  04/rsl/gram"
• 
  xmlnsxsi"http//www.w3.org/2001/XMLSchema-instan
  ce"
• 
  xsischemaLocation"
• 
  http//www.globus.org/namespaces/2003/04/rsl
• 
  c/ogsa-3.0/schema/base/gram/rsl.xsd
• 
  http//www.globus.org/namespaces/2003/04/rsl/gram
• 
  c/ogsa-3.0/schema/base/gram/gram_rsl.xsd"gt
• ltgramjobgt
• 
  ltgramexecutablegt ltrslpathgt
• 
  ltrslstringElement value"/bin/echo"/gt
  lt/rslpathgt
• 
  lt/gramexecutablegt
• 
  ltgramdirectorygt ltrslpathgt
• 
  ltrslstringElement value"/bin"/gt lt/rslpathgt
• 
  lt/gramdirectorygt
• 
  ltgramargumentsgt
• 
  ltrslstringgt ltrslstringElement value"Hello
  World"/gt lt/rslstringgt
• 
  lt/gramargumentsgt

• ((executable/bin/echo)
• (directory"/bin")
• (arguments"Hello World")
• (stdin/dev/null)
• (stdout"stdout")
• (stderr"stderr")
• (count1)
• )

43
Running Job(Assignment 3)

• GT3 command
• managed-job-globusrun -factory factoryservicename
  -file xmlfile
• where
• factoryservicename specified job service to
  process request
• filename specifies xml file containing RSL-2
  specification of job

44
Starting a job

• Master Managed Job Factory Service (MMJFS) needed
  to submit job.

45
Invoking MMJFS

• Invoke MMJFS with
• managed-job-globusrun
• and arguments -- named master job factory
  service to process job and an xml file to specify
  job.
• Command equivalent to GT 2 globusrun command.

46
Example

• user GLOBUS-LOCATION grid-proxy-init
• user GLOBUS-LOCATION managed-job-globusrun
  -factory http//terra8080/ogsa/services/base/gram
  /
• MasterForkManagedJobFactoryService -file
  GLOBUS_LOCATION/etc/test.xml
• This command causes the service to process
  test.xml - an xml document that contains the
  description of the job.

47
More details and practice

• See Assignment 3

48
More Information

• On-line
• Globus Toolkit 3.0 Quick Start, IBM redbooks,
  Sept 2003.
• http//redbooks.ibm.com/redbooks/pdf/sg246895.pdf
  
• http//redbooks.ibm.com/redpapers/pdfs/redp3697.pd
  f

49

• Introduction to the Globus ToolKittm, Resource
  Management Services, Nov. 6, 2003.
• http//www.globus.org
• The Master Managed Job Factory Service (MMJFS)
  by V. Silva, May 25, 2004.
• http//www-106.ibm.com/developerworks/
• grid/library/gr-factory/?cadgr-lnxw961GridMMJFS
• http//grid.hpctools.uh.edu/6397