Globus GRAM for Developers

1
Globus GRAM for Developers

• Stuart Martin, Peter Lane
• Argonne National Lab

2
Session Overview

• Q What is this session about?
• AThis presentation will cover the features,
  interface, architecture, performance, and future
  plans of the Globus Toolkit v4 Web Services Grid
  Resource Allocation and Management (GRAM4)
  component.
• Four-part discussion ( 20 mins/each)
• Overview of GRAM Model
• How to use client software
• How to administer servers
• Future plans

3
GRAM Part 1

• Overview of GRAM Model

4
What is GRAM?

• GRAM is a Globus Toolkit component
• For Grid job management
• GRAM is a unifying remote interface to Resource
  Managers
• Yet preserves local site security/control
• GRAM is for stateful job control
• Reliable operation
• Asynchronous monitoring and control
• Remote credential management
• File staging via RFT and GridFTP

5
Grid Job Management Goals

• Provide a service to securely
• Create an environment for a job
• Stage files to/from environment
• Cause execution of job process(es)
• Via various local resource managers
• Monitor execution
• Signal important state changes to client
• Enable client access to output files
• Streaming access during execution

6
Job Submission Model

• Create and manage one job on a resource
• Submit and wait
• Not with an interactive TTY
• File based stdin/out/err
• Supported by all batch schedulers
• More complex than RPC
• Optional steps before and after submission
  message
• Job has complex lifecycle
• Staging, execution, and cleanup states
• But not as general as Condor DAG, etc.
• Asynchronous monitoring

7
Job Submission Options

• Optional file staging
• Transfer files in before job execution
• Transfer files out after job execution
• Optional file streaming
• Monitor files during job execution
• Optional credential delegation
• Create, refresh, and terminate delegations
• For use by job process
• For use by GRAM to do optional file staging

8
Job Submission Monitoring

• Monitor job lifecycle
• GRAM and scheduler states for job
• StageIn, Pending, Active, Suspended, StageOut,
  Cleanup, Done, Failed
• Job execution status
• Return codes
• Multiple monitoring methods
• Simple query for current state
• Asynchronous notifications to client

9
Secure Submission Model

• Secure submit protocol
• PKI authentication
• Authorization and mapping
• Based on Grid ID
• Further authorization by scheduler
• Based on local user ID
• Secure control/cancel
• Also PKI authenticated
• Owner has rights to his jobs and not others

10
Secure Execution Model

• After authorization
• Execute job securely
• User account sandboxing of processes
• According to mapping policy and request details
• Initialization of sandbox credentials
• Client-delegated credentials
• Adapter scripts can be customized for site needs
• AFS, Kerberos, etc
• Multiple levels of audit possible
• Container
• Sudo
• Local scheduler

11
Secure Staging Model

• Before and after sandboxed execution
• Perform secure file transfers
• Create RFT request
• To local or remote RFT service
• PKI authentication and delegation
• In turn, RFT controls GridFTP
• Using delegated client credentials
• GridFTP
• PKI authentication
• Authorization and mapping by local policy files
• further authorization by FTP/unix perms

12
Users/Applications Job Brokers, Portals,
Command line tools, etc.
GRAM WSDLs Job Description Schema (executable,
args, env, )
WS standard interfaces for subscription,
notification, destruction
GRAM4
Resource Managers PBS, Condor, LSF, SGE,
Loadleveler, Fork
13
GRAM4 Approach
14
Other Approach Highlights

• Scalability improvements
• (discussed next)
• sudo/auth_and_exec
• to limit damage risk from software failures
• to improve audit capabilities
• Extensibility
• Retain scheduler adapter structure
• To extend for new platforms
• Improved authorization callouts
• To better integrate with site practices

15
Usage Scenarios the Ideal

• GRAM should add little to no overhead compared
  to an underlying batch system
• Submit as many jobs to GRAM as is possible to the
  underlying scheduler
• Goal - 10,000 jobs to a batch scheduler
• Goal efficiently fill the process table for
  fork scheduler
• Submit/process jobs as fast to GRAM as is
  possible to the underlying scheduler
• Goal - 1 per second

16
Usage Scenarios the Attempt

• Efforts and features towards the goal
• Allow job brokers the freedom to optimize
• E.g. Condor-G is smarter than globusrun-ws
• Protocol steps made optional and shareable
• Reduced cost for GRAM service on host
• Single WSRF host environment
• Better job status monitoring mechanisms
• More scalable/reliable file handling
• GridFTP and RFT instead of globus-url-copy
• Removal of non-scalable GASS caching

17
Production Quality

• Service performance
• Throughput
• Number of jobs (/bin/date) GRAM can process per
  minute
• 100
• Max concurrency
• Total jobs a GRAM service can manage at one time
  without failure
• 32,000
• Job burst
• Many simultaneous job submissions
• Are the error conditions acceptable?
• Job should be rejected, before overloading the
  service container or service host

18
Production Quality

• Service Stability Recovery
• Service uptime
• Under a moderate load, how long can the GRAM
  service process jobs without failure / reboot?
• Job recovery
• After reboot, processing/monitoring resumes for
  submitted jobs
• Clients resume control of jobs

19
Reasonable Applications Today

• High throughput job sets two approaches
• Use GRAM for every application task
• Jobs durations gt 1 minute
• Use GRAM for starting user/VO services
• Course-grain jobs handle task/transaction flow
• As in Condor glide-ins
• MPICH-G4 (MPIG)
• Large-scale multi-site/grid MPI jobs
• Co-allocation but no co-reservation yet
• Estimated release - Q4 2006

20
GRAM Part 2

• How to use client software

21
How to use Client Software

• Command line programs
• WSDL interface

22
Command Line Programs

• globusrun-ws
• Submit and monitor gram jobs
• grid-proxy-init
• Creates client-side user proxy
• wsrf-query
• Query a services resource properties
• globus-url-copy
• Transfer files to remote hosts
• globus-credential-delegate
• globus-credential-refresh
• Credential management to remote hosts

23
globusrun-ws

• Written in C (C WS Core)
• Faster startup and execution
• Supports GRAM multi-jobs or single jobs
• Submission, monitoring, cancellation
• Credential management
• Automatic or user-supplied delegation
• Streaming of job stdout/err during execution
• Advanced use of GridFTP client library

24
Simple Job Step 1

• Create a user proxy
• Your temporary grid credential
• Command Example
• grid-proxy-initYour identity/DCorg/DCdoegr
  ids/OUPeople/CNStuart Martin 564728Enter GRID
  pass phrase for this identityCreating
  proxy......................... DoneYour proxy is
  valid until Fri Jan 7 213531 2005

25
Simple Job Step 2

• Submit job to a GRAM service
• default factory EPR
• generate job RSL to default localhost
• Command example
• globusrun-ws -submit -c /bin/touch
  touched_itSubmitting job...Done.Job ID
  uuid002a6ab8-6036-11d9-bae6-0002a5ad41e5Terminat
  ion time 01/07/2005 2255 GMTCurrent job state
  ActiveCurrent job state CleanUpCurrent job
  state DoneDestroying job...Done.

26
Complete Factory Contact

• Override default EPR
• Select a different host/service
• Use contact shorthand for convenience
• Relies on proprietary knowledge of EPR format!
• Command example
• globusrun-ws -submit F \https//140.221.65.19
  34444/wsrf/services\/ManagedJobFactoryService
  \-c /bin/touch touched_it

27
Read RSL from File

• Command
• globusrun-ws -submit -f touch.xml
• Contents of touch.xml file
• ltjobgt ltexecutablegt/bin/touchlt/executablegt
  ltargumentgttouched_itlt/argumentgtlt/jobgt

28
Batch Job Submissions

• globusrun-ws -submit -batch -o job_epr -c
  /bin/sleep 50Submitting job...Done.Job ID
  uuidf9544174-60c5-11d9-97e3-0002a5ad41e5Terminat
  ion time 01/08/2005 1605 GMT
• globusrun-ws -monitor -j job_eprjob state
  ActiveCurrent job state CleanUpCurrent job
  state DoneRequesting original job
  description...Done.Destroying job...Done.

29
Batch Job Submissions

• globusrun-ws -submit -batch -o job_epr -c
  /bin/sleep 50Submitting job...Done.Job ID
  uuidf9544174-60c5-11d9-97e3-0002a5ad41e5Terminat
  ion time 01/08/2005 1605 GMT
• globusrun-ws -status -j job_eprCurrent job
  state Active
• globusrun-ws -status -j job_eprCurrent job
  state Done
• globusrun-ws -kill -j job_eprRequesting
  original job description...Done.Destroying
  job...Done.

30
Common/useful options

• globusrun-ws -J
• Perform delegation as necessary for job
• globusrun-ws -S
• Perform delegation as necessary for jobs file
  staging
• globusrun-ws -s
• Stream stdout/err during job execution to the
  terminal
• globusrun-ws -self
• Useful for testing, when you have started the
  service using your credentials instead of host
  credentials

31
Staging job

• ltjobgtltexecutablegt/bin/echolt/executablegtltdirector
  ygt/tmplt/directorygtltargumentgtHellolt/argumentgtltstd
  outgtjob.outlt/stdoutgtltstderrgtjob.errlt/stderrgtltfil
  eStageOutgt lttransfergt ltsourceUrlgtfile///tmp
  /job.outlt/sourceUrlgt ltdestinationUrlgt
  gsiftp//host.domain2811/tmp/stage.out
  lt/destinationUrlgt lt/transfergtlt/fileStageOutgt
• lt/jobgt

32
RFT Options

• ltfileStageOutgtlttransfergt ltsourceUrlgtfile///tmp
  /job.outlt/sourceUrlgt ltdestinationUrlgt
  gsiftp//host.domain2811/tmp/stage.out
  lt/destinationUrlgt
• ltrftOptionsgt
• ltsubjectNamegt /DCorg/DCdoegrids/OUPeople/
  CNStuart Martin 564728
• lt/subjectNamegt
• ltparallelStreamsgt4lt/parallelStreamsgt
• lt/rftOptionsgt
• lt/transfergt
• lt/fileStageOutgt

33
RSL Variable

• Enables late binding of values
• Values resolved by GRAM service
• System-specific variables
• GLOBUS_USER_HOME
• GLOBUS_LOCATION
• GLOBUS_SCRATCH_DIR
• Alternative directory that is shared with compute
  node
• Typically providing more space than users HOME
  dir

34
RSL Variable Example

• ltjobgtltexecutablegt/bin/echolt/executablegtltargument
  gtHOME is GLOBUS_USER_HOMElt/argumentgtltargumentgt
  SCRATCH GLOBUS_SCRATCH_DIRlt/argumentgtltargume
  ntgtGL is GLOBUS_LOCATIONlt/argumentgtltstdoutgtG
  LOBUS_USER_HOME/echo.stdoutlt/stdoutgtltstderrgtGL
  OBUS_USER_HOME/echo.stderrlt/stderrgt
• lt/jobgt

35
RSL Extensions Support

• 4.0.3 does not support extension by default
• Update packages are available to add extension
  support
• http//www.globus.org/toolkit/downloads/developmen
  t/
• globus_gram_job_manager-7.14 plus dependencies
• All 4.1.x releases support extensions by default

36
RSL Extensions Example

• ltjobgtltexecutablegt/bin/echolt/executablegtltextensio
  nsgt ltemail_addressgtjoeshmo_at_gmail.comlt/email_addre
  ssgtltextensionsgt
• lt/jobgt

• Simple string extension elements are converted
  into single-element arrays
• Code example in pbs.pmif(description-gtemail_add
  ress() ne '') print JOB 'PBS -M ', \
  description-gtemail_address(), "\n"

37
How to use Client Software

• Command line programs
• WSDL interface

38
ManagedJobFactory portType

• createManagedJob operation
• Creates either an MMJR or MEJR
• Input
• Initial Termination Time
• Job ID
• UUID of the job resource, for job
  reliability/recoverability
• Subscribe Request
• Client can include a request to subscribe for job
  state notifications with the job submission to
  avoid an extra operation call
• Job Description / RSL
• Either a single or multi-job description
• Output
• newTerminationTime - new termination time of the
  job resource
• managedJobEndpoint - EPR of the newly created job
  resource
• subscriptionEndpoint - EPR of the notification
  subscription

39
ManagedJob portType

• Base port type for the MEJS and MMJS
• release operation
• Release a holdState set in the job description
• Only one hold state can be set/released
• Input None
• Output None
• State change notifications
• State - job state (Active, Pending, Done,
  Cleanup)
• Fault - fault causing a Failed state (if
  applicable)
• Exit Code - exit code of the job process
• Holding - boolean indicating if the job is in a
  hold state

40
ManagedJob portType

• On destroy, or soft state termination
• The MJS will cleanup everything
• Stop any outstanding tasks
• Cancel/terminate the execution
• Destroy RFT stage in, out requests
• Process CleanUp state
• Submit request to RFT to remove files/directories
• RSL attribute fileCleanUp
• Remove job user proxy file
• Destroy job resource

41
ManagedExecutableJobService

• Executes the requested job process(es) specified
  in the RSL
• Resource Properties (ManagedExecutableJobPortType)
  
• serviceLevelAgreement - the RSL / Job Description
• state - the current job state
• faults - the fault causing a Failed state
• localUserId - the username of the resource owner
• userSubject - the GSI subject of the resource
  owner
• holding - boolean indiciating the job is holding
• stdoutURL - the GridFTP URL to the stdout file
• stderrURL - the GridFTP URL to the stderr file
• credentialPath - the local path to the user proxy
  file
• exitCode - the exit code of the job proces (if
  applicable)

42
ManagedMultiJobService

• Processes a multi-job RSL
• submits the sub-jobs to the specified
  ManagedJobFactoryService.
• Sub-jobs cannot be multi-jobs themselves.
• Resource Properties (ManagedMultiJobPortType)
• serviceLevelAgreement - the multi-job RSL / Job
  Description
• state - the current overall state
• faults - the fault causing a Failed state
• localUserId - the username of the resource owner
• userSubject - the GSI subject of the resource
  owner
• holding - boolean indiciating all jobs are
  holding
• subJobEndpoint - list of endpoints to the
  sub-jobs

43
Our Goals

• Highly functional interface
• grid service WSDLs
• C API
• Java API
• Expressive job description language
• Basic command line clients
• Should be useable from shell scripts
• Collaborate with others to create more capable
  and complete clients
• E.g. Condor-G, TGs Science Gateways, Portals

44
GRAM Part 3

• How to administer servers

45
4.0 Quickstart Guide

• Consult this guide first for basic GT setup
• Setting up first machine
• Setting up second machine
• Setting up a compute cluster - PBS
• www.globus.org/toolkit/docs/4.0/admin/docbook/qui
  ckstart.html
• Then consult GRAM admin guide for additional
  details
• www.globus.org/toolkit/docs/4.0/admin/docbook/ch11
  .html

46
Typical GRAM service setup

• Host credentials
• For client/service authentication
• For client authorization of the service
• Existing GT2/GT3 host certs can be used
• Gridmap file
• Entries for each user allowed to execute jobs
• Maps the grid ID to a local user account
• Same syntax as GT2, GT3 gridmap files
• Installed sudo
• Method for GRAM to runs commands in the users
  account

47
sudo configuration

• sudo policies
• Done by hand by rootRunas_Alias GRAMUSERS !
  root, ! wheel,
• globus ALL(GRAMUSERS) NOPASSWD
  /sandbox/globus/install/libexec/globus-gridmap-and
  -execute /sandbox/globus/install/libexec/globus
  -job-manager-script.pl globus ALL(GRAMUSERS)
  NOPASSWD /sandbox/globus/install/libexec/globu
  s-gridmap-and-execute /sandbox/globus/install/l
  ibexec/globus-gram-local-proxy-tool
• globus-gridmap-and-execute
• Redundant if sudo is locked down tightly
• Enforce that GRAM only targets accounts in
  gridmap
• So sudo policy need not enumerate all GRAM users
  at large/dynamic sites
• In fact, you can audit this tool and change
  GRAMUSERS to ALL if you like
• Replace this with your own authz tool (callout)

48
Local Resource Manager Adapters

• GT provides/supports 4 RM adapters
• PBS, LSF, Condor, Fork
• 3rd party RM adapters exist
• SGE, LoadLeveler, GridWay
• Tell us about yours and well add to GT web
  pages!
• All 4 RM adapters are included in all binary and
  source installers
• Only Fork is configured automatically
• Configuring an RM adapter
• Add configure arguments
• ./configure --enable-wsgram-pbs

49
File staging functionality

• GridFTP Server
• Could be run on a separate host from GRAM service
  container to improve performance / scalability
• cpu intensive
• globus_gram_fs_map_config.xml
• Config the GridFTP server(s) to use for local
  file staging
• RFT
• Requires PostgreSQL DB setup
• Usability 4.1.x Defaults to embedded DB (Derby)

50
GRAM / GridFTP file system mapping

• Associates compute resources and GridFTP servers
• Maps shared filesystems of the gram and gridftp
  hosts, e.g.
• Gram host mounts homes at /pvfs/home
• gridftp host mounts same at /pvfs/users/home
• GRAM resolves file/// staging paths to local
  GridFTP URLs
• File///pvfs/home/smartin/file1... resolves to
• gsiftp//host.domain2811/pvfs/users/home/smartin/
  file1
• GL/etc/gram-service/globus_gram_fs_map_config.xml
  
• Client will need to know mappings to stage files
  separately from WS GRAM

51
Non-default Setup

• ./setup-gram-service-common
• To change GRAM configuration
• Run in GLOBUS_LOCATION/setup
• GridFTP Server config
• Default is for localhost, port 2811
• --gridftp-servergsiftp//gridftp.host.org1234
• RFT Service config
• Default is localhost, port 8443
• --stage-protocolhttps
• --staging-hosthost.domain.org
• --staging-port4321

52
Setup Container Credentials

• Default host credentials
• /etc/grid-security/containercert.pem
• /etc/grid-security/containerkey.pem
• To configure for a user proxy
• Update container global security descriptor
• Comment out ltcredentialgt element
  GL/etc/globus_wsrf_core/global_security_descripto
  r.xml
• Tell GRAM the subject to expect for authorization
  of the RFT service
• ./setup-gram-service-common --staging-subject
  "/DCorg/DCdoegrids/OUPeople/CNStuart Martin
  564720
• Use -self argument with globusrun-ws
• Default GT auth in 4.1.1 will be host or
  self

53
GRAM Part 4

• Future Plans

54
4.2 Series WS GRAM

• 4.1.x is dev series for eventual stable 4.2.x
  stable series
• 4.1.0 released July 06
• RSL extension support
• globus-job--ws scripts included by default
• Improved service throttling controls
• Persistence data stored in DB
• resource manager adapter API
• Removed unnecessary dependencies to Pre-WS GRAM
• 4.1.1 (no target date yet)
• Initial support for JSDL jobs
• Service auditing to DB

55
WS GRAM Standards Compliance

• JSDL
• Target is 4.1.1 (definitely 4.2.0)
• Will preserve current interface, so 4.0.x job
  descriptions will work just fine
• Adding new createManagedJobFromJSDLDocument
  operation
• Globusrun-ws will choose appropriate create
  operation based on job description contents
• OGSA-BES
• Target is 4.4 (spec is not finished, so 4.2 is
  unlikely)
• Will preserve 4.0.x interface as well

56
Service Auditing

• Follow along on bugzilla roadmap item
• http//bugzilla.globus.org/bugzilla/show_bug.cgi?i
  d4409
• Add yourself to cc list
• Prototype written and deployed on TeraGrid
• In evaluation phase
• provides the capability for a TG grid user to get
  TG usage info using a grid job id (from GRAM)
• Audit DB entries provide join between grid job id
  and local TG accounting DB
• Will be included in 4.1.x series to be included
  in 4.2
• Probably disable by default in GT releases

57
Advanced Reservation

• Investigation is underway
• No firm plans yet, but high on our priority list
• Follow along on bugzilla roadmap item
• http//bugzilla.globus.org/bugzilla/show_bug.cgi?i
  d4045

58
Performance testing with OSG

• Test scenario
• submit large (3500) job run through condor-g to
  WS GRAM to LRM condor
• Job is create unique job dir 2MB stageIn, 2MB
  stageOut, cleanup job dir
• Solved reliability issue with default condor-g
  jobs
• Included in 4.0.3
• Found/fixed bugs in RFT which effected
  performance by appox 250 for staging jobs
• From 5.2 jpm to 13 jpm
• Patches to 4.0.3 will be made available soon
• We plan on writing up results and provide config
  recommendation for GT container and condor-g

59
WS GRAM Usage Statistics

• July 6 thru Aug 6th 2006
• 651517 jobs submitted
• 25 unique domains (e.g. .edu, .org, .gov)
• 356 unique IPs (Container installations with WS
  GRAM)

60
Documentation

• 4.0.x GRAM documentation
• Guides admin, user, developer, overview, public
  interface
• http//www.globus.org/toolkit/docs/4.0/execution/w
  sgram/
• 4.1.x GRAM documentation
• http//www.globus.org/toolkit/docs/4.1/execution/w
  sgram/
• Main 4.0.x documentation
• http//www.globus.org/toolkit/docs/4.0/
• Download, release notes, links to all GT
  projects/ components

61
Writing New RM Adapters

• http//www.globus.org/toolkit/docs/4.0/execution/w
  sgram/developer/scheduler-tutorial.html
• Scheduler perl modules (e.g. pbs.pm)
• Submitting jobs, canceling jobs, setup and
  packaging
• Scheduler Event Generator (SEG)
• Monitoring events from the scheduler for all job
  for all users it runs under a privileged account

62
Bugzilla

• If youve found a bug (not a question!)
• http//bugzilla.globus.org/
• GRAM product, wsrf components

63
Globus Development

• GlobDev - Open development
• Globus governance model based on Apache
• Developers (committers) control direction of
  software components (projects)
• http//dev.globus.org
• GRAM project
• http//dev.globus.org/wiki/GRAM
• Email lists gram-user, gram-dev, gram-announce,
  gram-commit
• GT project
• gt-user, gt-dev

64
Thanks to the GRAM developers!

• Peter Lane - ANL
• Joe Bester - ANL
• Ravi Madduri - ANL
• Martin Feller - UofC
• Plus the entire GT dev team

65
Meet the Developers Session at Globus Alliance
Booth (152A-P7)

• September 12800am - 900am "Java WS Core and
  Security (C, Java)"  -- Olle Mulmo, Jarek Gawor,
  Rachana Anantakrishnan
• 1130am -1230pm "RLS" -- Rob Schuler, Ann
  Chervenak1230pm -130pm "MDS" -- Mike D'arcy,
  Laura Pearlman300pm - 400pm Resource
  Management (GRAM, Virtual Workspaces and Dynamic
  Accounts)" Stu Martin, Peter Lane, Tim Freeman,
  Kate Keahey600pm - 700pm "C WS Core" -- Joe
  Bester700pm - 800pm "Python WS Core" -- Joshua
  Boverhof
• September 13800am - 900am "GridShib" -- Von
  Welch, Ton Scavo, Tim Freeman
• 1130am - 1230pm "GT Installation and
  Administration" -- Charles Bacon1230pm - 130pm
  "MyProxy" -- Jim Basney300pm - 400pm "GridFTP,
  XIO, RFT" -- John Bresnahan, Ravi Madduri

66
COME CELEBRATE WITH US!
In appreciation of your support of all things
Globus over the past decade, you are cordially
invited to the Globus 10th Birthday Party.
When Monday, September 11, 2006 - 700pm,
immediately following Ian Fosters Globus State
of the Union Keynote.
Where The convention center concourse, in the
center of the GlobusWORLD / GridWorld conference
activity.
What Food, drinks, music, friends and lots of
fun!