Presentation Title Presentation Subtitle

1
Hands-On SessionUsing Grid
Regional SEE-GRID-SCI Meteo Training Faculty of
Electrical Engineering Banja Luka July 2-3, 2009
Vladimir Slavnic Institute of Physics,
Belgrade Serbia slavnic_at_scl.rs
The SEE-GRID-SCI initiative is co-funded by the
European Commission under the FP7 Research
Infrastructures contract no. 211338
2
Setting up the user account
3
Setting up the user accountUser Interface (UI)?

• Access point to Grid
• User must have a local account on machine
• Basic grid operations
• list resources suitable for job
• submit jobs
• cancel jobs
• retrieve job output
• show job status
• file operations

4
Setting up the user account Certificates

• Grid passport
• X.509
• Subject Name (SN)?
• /CRS/OAEGIS/OUInstitute of Physics
  Belgrade/CNVladimir Slavnic
• Certification Authority (CA)?
• Obtaining a certificate
• grid-cert-request
• userkey.pem usercert_request.pem usercert.pem
• Virtual Organisation (VO)?

5
Setting up the user accountSetting up

• .globus
• Right permissions
• 444 usercert.pem
• 400 userkey.pem
• Checking a certificate
• grid-cert-info
• grid-cert-info -subject (-enddate) (-issuer)?
• openssl verify -CApath \
• /etc/grid-security/certificates/
  /.globus/usercert.pem
• grid-proxy-init -verify

6
Setting up the user account Proxies

• Standard proxies
• grid-proxy-init
• grid-proxy-info
• grid-proxy-destroy
• VOMS proxies
• Group
• Role
• voms-proxy-init -voms ltvogt
• voms-proxy-init voms ltaliasgtltgroup namegt
• Roleltrole namegt
• voms-proxy-info (-all)?

7
Setting up the user accountProxy renewal

• MyProxy server
• echo MYPROXY_SERVER
• myproxy-init -s ltmyproxy_servergt -d -n
• myproxy-info -s ltmyproxy_servergt -d
• myproxy-destroy -s ltmyproxy_servergt -d

8
Information service
9
INFORMATION SERVICElcg-infosites lcg-info

• lcg-infosites --vo ltvogt ltoptiongt -v ltverbositygt
  -f ltsitegt --is ltbdiigt
• lcg-infosites --vo seegrid ce
• lcg-infosites --vo seegrid se
• lcg-infosites --vo seegrid closeSE
• lcg-info list-attrs
• lcg-info --list-se --query
  'SEse.csk.kg.ac.yu' --attrs CloseCE

10
INFORMATION SERVICEldap

• ldap search examples
• RESOURCE
• ldapsearch -x -H ldap//ce64.phy.bg.ac.yu2170/
  -b Mdsvo-nameresource,oGrid
• sBDII
• ldapsearch -x -H ldap//ce64.phy.bg.ac.yu2170
  -b mds-vonameAEGIS01-PHY-SCL,ogrid
• top-level BDII
• ldapsearch -x -H ldap//bdii.phy.bg.ac.yu2170/
  -b 'Mds-vonamelocal,oGrid'

11
WatG Browser

• Web based Grid Information System browser
• Visualization of BDII information
• Highly responsive tool because it supports
• partial refreshes and desynchronization of a
• web page.
• Developed with Google Web Toolkit (GWT)?
• open source Java software development framework.
• '

12
WatG Browser
http//watgbrowser.scl.
rs8080/
13
Data management
14
DATA MANAGEMENT

• Storage element
• Classic SE
• LCG Disk pool manager
• CASTOR
• Storage Resource Manager (SRM)?
• LFC Concept
• lfc and lcg commands

15
DATA MANAGEMENTGrid file

• LFN Logical File Name
• GUID - Grid Unique Identifier
• SURL Storage URL

16
DATA MANAGEMENTLFC scheme
17
DATA MANAGEMENTLFC commands

• echo LFC_HOST
• lfc-ls - List file/directory entries in a
  directory
• lfc-mkdir - Create directory
• lfc-ln - Make a symbolic link to a
  file/directory
• lfc-ls - List file/directory entries in a
  directory
• lfc-chmod- Change access mode of a LFC
  file/directory
• lfc-chown - Change owner and group of a LFC
  file/directory
• lfc-getacl - Get file/directory access control
  lists
• lfc-setacl - Set file/directory access control
  lists

18
DATA MANAGEMENTLFC commands examples (1)?

• Listing the entries of a LFC directory
• lfc-ls /grid/seegrid/vlada
• new_file
• result
• Creating directories in the LFC
• lfc-mkdir /grid/seegrid/vlada/new_dir
• lfc-ls -l /grid/seegrid/vlada
• drwxrwxr-x 0 165 101 0 Feb 19 0030 new_dir
• -rw-rw-r-- 1 165 101 27 Feb 18 0108 new_file
• -rw-rw-r-- 1 165 101 83 Feb 15 0030 result

19
DATA MANAGEMENTLFC commands examples (2)?

• Creation of symbolic links
• lfc-ln -s /grid/seegrid/vlada/result \
• /grid/seegrid/vlada/result_link
• lfc-ls -l /grid/seegrid/vlada/result_link
• lrwxrwxrwx 1 165 101 0 Feb 19 0042
• /grid/seegrid/vlada/rezult_link -gt
  /grid/seegrid/vlada/rezult
• Adding metadata information to LFC entries
• lfc-setcomment /grid/seegrid/vlada/result
  "best result"
• lfc-ls --comment /grid/seegrid/vlada/result
• /grid/seegrid/vlada/result best result

20
DATA MANAGEMENTlcg_utils-Replica Management

• echo LCG_GFAL_INFOSYS
• lcg-cp Copies a Grid file to a local destination
• (download)?
• lcg-cr Copies a file to a SE and registers the
  file in the
• catalogue (upload)?
• lcg-del Deletes one file (either one replica or
  all
• replicas)?
• lcg-rep Copies a file from one SE to another SE
  and
• registers it in the catalogue (replicate)?
• lcg-gt Gets the TURL for a given SURL and
  transfer
• protocol

21
DATA MANAGEMENTlcg_utils-File Catalogue
Interaction

• lcg-aa Adds an alias in the catalogue for a given
  GUID
• lcg-ra Removes an alias in the catalogue for a
  given
• GUID
• lcg-rf Registers in the catalogue a file
  residing on an
• SE
• lcg-uf Unregisters in the the catalogue a file
  residing on an SE
• lcg-la Lists the aliases for a given LFN, GUID
  or SURL
• lcg-lr Lists the replicas for a given LFN, GUID
  or SURL
• lcg-ls Lists file information for given SURLs or
  LFNs

22
DATA MANAGEMENTlcg_utils examples(1)?

• Upload a file
• lcg-cr file/home/vlada/banja_luka/mpi.jdl
• guid830e7a82-f9fb-4c04-aafd-e24a5f25acdb
• specify lfn
• lcg-cr -l lfn/grid/seegrid/vlada/mpi \
• file/home/vlada/banja_luka/mpi.jdl
• guid25fdcd16-bc76-4458-9c1a-8c597fb33194
• Replicate a file (one replica per SE)
• lcg-rep -d grid15.rcub.bg.ac.yu \
  lfn/grid/seegrid/vlada/mpi

23
DATA MANAGEMENTlcg_utils examples(2)?

• List replicas
• lcg-lr lfn/grid/seegrid/vlada/mpi
• srm//grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/
  ae
• gis/generated/2009-02-18/file8f99889c-c00a-4b7b-
• 89d3-d9abf66ba103
• srm//se.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/
  generated/2009-02-18/file576fbae0-ba6a-4b21-a205-3
  d115b2cf7fb
• List guid
• lcg-lg lfn/grid/seegrid/vlada/mpi
• guid25fdcd16-bc76-4458-9c1a-8c597fb33194

24
DATA MANAGEMENTlcg_utils examples(3)?

• Listing files and directories
• lcg-ls -l lfn/grid/seegrid/vlada/mpi
• -rw-rw-r-- 1 165 101 349 lfn/grid/seegrid/vlad
  a/mpi
• Copying files out of the Grid (download)?
• lcg-cp lfn/grid/seegrid/vlada/mpi \
  file/home/vlada/banja_luka/down/mpi
• Deleting replicas
• lcg-del srm//grid15.rcub.bg.ac.yu/dpm/rcub.bg.
  ac.yu/home/aegis/generated/2009-02-19/file36d25d0d
  -ee61-477eb667-40b3791786a4
• lcg-del -a lfn/grid/aegis/vlada/mpi

25
Data Management Web Portal

• Developed by Belgrade University Computer Centre
• Usage
• Data Management
• Logical Files
• Development Environments and Portals
• Description
• Enables easy interaction with grid file catalogue
  using simple Web-based user interface.
• Based on SEE-GRID File Management Java API.
• Can be integrated with other Web applications.

26
Data Management Web Portal

• Available features
• Directory browsing
• Directory management operations creating new
  subdirectories, renaming/moving and deleting
  directories, replicating directory content,
  reading directory attributes and permissions.
• Copying and registering of new files to the
  catalog as well as file deletion.
• Reading file attributes and permissions.
• Replica alias management operations.
• Upload and download of files with possible
  integration of application-defined upload handlers

27
Data Management Web Portal

• http//grid02.rcub.bg.ac.yu8080/NewDataManagement
  /login-REFS.jsp
• http//grid02.rcub.bg.ac.yu8080/NewDataManageme
  nt/login-WRF-ARW.jsp

28
Job operations
29
JDL

• JDL (Job Description Language) file
• attribute expression
• Executable "test.sh"
• Arguments "fileA fileB"
• StdOutput "std.out"
• StdError "std.err"
• InputSandbox "test.sh", "fileA", "fileB"
• OutputSandbox "std.out", "std.err"
• Environment AEGIS_PATHHOME/AEGIS"
• Requirements RegExp("ce64.phy.bg.ac.rs",other
  .GlueCEUniqueID)

30
SUBMITING JOBS

• Generating proxy
• voms-proxy-init -voms ltVO_NAMEgt
• Myproxy
• myproxy-init -d n -s ltmyproxy_servergt
• Delegate proxy to WMS
• glite-wms-job-delegate-proxy -d ltdelegIDgt
• Job matching
• glite-wms-job-list-match -a test.jdl
• Job submission
• glite-wms-job-submit -d ltdelegIDgt -i
  ltjob_id_filegt test.jdl

31
MONITORING JOBS

• View status of job
• glite-wms-job-status ltjob_IDgt
• glite-wms-job-status -i ltjob_id_filegt
• View logging details
• glite-wms-job-logging-info -v
  ltverbosity_levelgt ltjob_IDgt
• glite-wms-job-logging-info -v
  ltverbosity_levelgt -i ltjob_id_filegt
• Verbosity level can be from 0 to 3.

32
CANCELING JOBS AND RETRIEVING OUTPUT

• Cancel a job
• glite-wms-job-cancel ltjob_IDgt
• glite-wms-job-cancel -i ltjob_id_filegt
• Retrieve files from Output Sandbox
• glite-wms-job-output ltjob_IDgt
• glite-wms-job-output -i ltjob_id_filegt
• glite-wms-job-output --dir ltdirectory_pathgt
  ltjob_IDgt

33
ADVANCED JOB TYPES

• Job Collection - submission of a set of jobs
  whose description files are placed in one
  directory
• Parametric Jobs - submission of a set of jobs
  having the same descriptions apart from the
  values of the parametric attributes. Use when
  your jobs differs only in argument values or
  input/output files.
• DAG jobs (Direct Acyclic Graphs) - submission of
  a set of jobs where the input, output, or
  execution of one or more jobs depends on one or
  more other jobs.
• Interactive Jobs
• MPI Jobs

34
JOB COLLECTION
Type "Collection" InputSandbox
(fileA) Requirements RegExp("ce64.phy.bg.ac.y
u",other.GlueCEUniqueID) nodes JobType
"normal" Executable "/bin/hostname" Arguments
"-i" StdOutput "out" InputSandbox
(root.InputSandbox, fileB) StdError
"err" OutputSandbox "out","err" , JobType
"Normal" Executable "/bin/date" StdOutput
"date" StdError "err" OutputSandbox
"date"
35
JOB COLLECTIONsubmission

• Collection defined in one file
• glite-wms-job-submit -a -o id collection.jdl
• Folder contains jdl files
• glite-wms-job-submit -a --collection jdl
• Collection status view
• Collection output
• glite-wms-job-output -i id --dir kolekcija
• ls kolekcija
• ids_nodes.map Node_0 Node_1

36
PARAMETRIC JOBS
Type "Job" JobType "Parametric" Executabl
e "/bin/echo" Arguments "_PARAM_" Paramete
rs red, green, blue StdOutput
"colour_PARAM_.txt" StdError
"error_PARAM_.txt" OutputSandbox
"colour_PARAM_.txt","error_PARAM_.txt"
37
PARAMETRIC JOBS
Type "Job" JobType Parametric"
Executable "exec" StdInput
"input_PARAM_.txt" StdOutput
"output_PARAM_.txt" Parameters
10 ParameterStart 1 ParameterStep
1 InputSandbox "ltexecgt", "ltparam_filegt" Ou
tputSandbox "output_PARAM_.txt"
38
DAG JOBS
Type "dag" InputSandbox "job.sh" Nodes
nodeA file "A.jdl"
nodeB file b.jdl" nodeC
file C.jdl" nodeD
Description JobType "Normal"
Executable "job.sh"
StdOutput "std.out" StdError
"std.err" InputSandbox
root.InputSandbox0 OutputSandbox
"std.out","std.err"
Dependencies
nodeA,nodeB,nodeA,nodeC,nodeB,nodeC,nodeD

39
MPI JOBS

• JDL file
• Type "Job"
• JobType "MPICH"
• NodeNumber 5
• Executable "mpi.sh"
• Arguments "mpi"
• StdOutput "mpi.out"
• StdError "mpi.err"
• InputSandbox "mpi.sh","mpi.c"
• OutputSandbox \ "mpi.err","mpi.out","mpiexec
  .out"

40
LINKS

• http//aegis-ca.rcub.bg.ac.yu/
• https//edms.cern.ch/file/722398//gLite-3-UserGuid
  e.pdf
• http//wiki.egee-see.org/index.php/Quick_User_Guid
  e_for_Submitting_Jobs
• https//edms.cern.ch/document/590869/1/
• http//wiki.egee-see.org/index.php/SEE-GRID_MPI_Us
  er_Guide
• http//wiki.egee-see.org
• http//wiki.egee-see.org/index.php/Data_Management
  _Web_Portal_-_User_Guide