UNIX

1
Getting Started with HPC On Iceberg
Michael Griffiths and Deniz Savas Corporate
Information and Computing Services The University
of Sheffield www.sheffield.ac.uk/wrgrid
2
Outline

• Introduction to Iceberg
• Getting connected
• Basic Linux
• Using the Linux Operating System
• Running Applications and Jobs on Iceberg
• How to distribute work across the cluster
• The N8 Facility
• Getting Help

3
Types of Grids

• Cluster Grid
• Computing clusters ( e.g. iceberg )
• Enterprise Grid, Campus Grid, Intra-Grid
• Departmental clusters, servers and PC network
• Cloud, Utility Grid
• Access resources over internet on demand
• Global Grid, Inter-grid
• White Rose Grid, National Grid Service, Particle
  Physics Data Grid

4
Iceberg Cluster
There are two head-nodes for the iceberg cluster

login
login
login
HEAD NODE1 Iceberg(1)
HEAD NODE2 Iceberg(2)
qsh,qsub,qrsh
qsh,qsub,qrsh
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
There are 232 worker machines in the cluster
All workers share the same user filestore
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
5
iceberg cluster specifications

• AMD-based nodes  containing
• 96 nodes each with 4 cores and 16 GB of memory
• 31 nodes each with 8 cores and 32 GB of memory
• TOTAL AMD CORES 632,   TOTAL MEMORY 2528
  GBThe 8-core nodes are connected to each other
  via 16 GBits/sec infiniband for MPI jobsThe
  4-core nodes are connected via the much slower "1
  Gbits/sec" ethernet connections for MPI jobs.
  Scratch space on each node is 400 GBytes
• Intel Westmere based nodes all infiniband
  connected, containing
• 103 nodes each with 12 cores and 24 GB of memory
  ( i.e. 2 6-core Intel X5650 )
• 4  nodes each with 12 cores and 48 GB of memory
• 8  Nvidia Tesla Fermi M2070s GPU units for GPU
  programming
• TOTAL INTEL CPU CORES 1152  , TOTAL MEMORY
  2400 GBScratch space on each node is 400
  GBTotal GPU memory 48 GBEach GPU unit is
  capable of about 1TFlop of single precision
  floating point performance, or 0.5TFlops at
  double precision. Hence yielding maximum GPU
  processing power of upto 8 TFlops in total.

6
Introduction to Linux on Iceberg

• What are UNIX and Linux?
• The Shell
• Getting Connected
• Basic UNIX/Linux Commands
• Working with Directories
• Editing files
• Help!
• Running Programs
• Exercise

7
What is UNIX/Linux ?

• Multi-Tasking O/S
• Multi-User O/S
• Available on a range of Computers

8
History of UNIX/Linux?

• .

• Unix operating system was developed around 1969
  in the Bell Labs
• Originally written using C
• Around 1990 Linus Torvalds of Helsinki University
  started off a freely available academic version
  of Unix
• Linux is the Antidote to a microsoft dominated
  future

9
Which UNIX?

• SunOS Sun Microsystems now OpenIndiana,
  OpenSolaris from Oracle
• IRIX Silicon Graphics
• HP-UX Hewlett Packard
• Linux For IBM PC compatibles
• There are a number of certification bodies with
  published standards and test suites to ensure
  quality of products. Such as
• Posix Portable Operating System Interface ( IEEE
  standard).

10
Linux and GNU

• Linux is an implementation of Unix
• Linux/Unix operating system is written in C
• Linux is not part of the GNU project but uses the
  same licensing agreements
• Many of the linux utilities and tools are taken
  from the GNU project.
• There are many flavours of linux distributions.
  The mix of the kernel (linux) with the utilities
  (GNU and other) and the installation procedure
  determine the flavour. Some of these are
• Fedora
• SuSE
• Redhat
• Debian
• Mandrake
• Knoppix

11
UNIX Internals (Simplified)

• Kernel
• System Internals
• Shell
• Command Interpreter
• Programming language
• File System
• Process Management

12
UNIX Shells

• sh Bourne Shell (Original Shell) (Steven Bourne
  of ATT)
• bash Bourne Again Shell (GNU Improved Bourne
  Shell)
• csh C-Shell (C-like Syntax)(Bill Joy of Univ. of
  California)
• ksh Korn-Shell (Bournesome C-shell)(David Korn
  of ATT)
• tcsh Turbo C-Shell (More User Friendly
  C-Shell).
• You can switch from one shell to another by just
  typing the name of the shell. exit return you
  back to previous shell.

13
Getting an account

• All staff and research students are entitled to
  use iceberg
• For Registration See
• http//www.shef.ac.uk/wrgrid/register
• Staff can have an account by simply emailing
  ucards-reg_at_sheffield.ac.uk

14
passwords

• In normal linux environment the passwd command
  can be used to change the user passwords.
  However, because we manage passwords centrally
  this command will not work on iceberg.
• If you wish to change your iceberg password you
  will have to do this via a web interface at the
  following URL
• http//www.shef.ac.uk/cics/password

15
Access iceberg Remote logging in

• Terminal access is described at
  http//www.shef.ac.uk/wrgrid/using/access
• Recommended access is via any browser at
  www.shef.ac.uk/wrgrid This uses Sun Global
  Desktop ( All platforms, Graphics-capable)Also
  possible
• Using an X-Windows client ( MS Windows,
  Graphics-capable)
• Exceed 3D
• Cygwin
• Various ssh clients (MS Windows, Terminal-only )
• putty, SSH
• Note ssh clients can also be used in combination
  with Exceed or Cygwin to enable graphics
  capability. Above web page describes how this can
  be achieved.

16
Access icebergfrom MAC or Linux platforms

• The web browser method of access ( as for Windows
  platforms) also works on these platforms.
• More customary and efficient method of access is
  by using the ssh command from a
  command-shell.Example ssh X
  iceberg_username_at_iceberg.shef.ac.ukNote1 -X
  parameter is needed to make sure that you can use
  the graphics or gui capabilities of the software
  on iceberg.Note2 Depending on the configuration
  of your workstation you may also have to issue
  the command xhost before the ssh command.

17
Basic X Concepts

• X Server runs on local machine
• PC Exceed, Cygwin, Xming
• UNIX Workstation Included in OS
• Apple Mac Exodus
• X Client runs on remote machine
• Graphical Application
• xterm
• xcalc
• Modelling and visualisation packages etc.

18
Multiple ssh or xterm shells

• There are no limits to the number of ssh or xterm
  windows one can start simultaneously by methods
  described in the previous slides. You may also
  start extra xterm windows from the host by simply
  typing xterm
• On iceberg we also have a local command named
  Xterm that starts up an xterm window with nicer
  to use parameters. On iceberg we strongly
  recommend that you use Qsh rather than xterm or
  Xterm command so as to make use of a free worker
  node. Qsh will act like Xterm but will make use
  of a worker node.
• Typing exit will terminate an xterm or ssh
  session neatly.
• This will also close the xterm window but not the
  ssh window. ssh windows can be closed via the
  file? exit menu.

19
Web browser method of access Sun Global Desktop
Start session on headnode
Start an interactive session on a worker
Start Application on a Worker
Help
20
Operating system and utilities

• Linux Operating System version is-
• Scientific Linux, which is RedHat Enterprise 5
  based
• Default Shell is- bash
• Available editors-Graphical Editors
• gedit ( not available on headnode)
• emacs
• nedit
• Text editors
• vi
• pico ( not available on headnode)
• nano

21
Login Environment

• Default shell is bash ( you can request to change
  your default shell ).
• On login into iceberg many environment variables
  are setup automatically to provide a consistent
  interface and access to software.
• Each user has a .bashrc file in their directory
  to help setup the user environment.
• Type set to get a list of all the environment
  variables.
• Change all variables that are in CAPITALS with
  extreme care.
• Modify/enhance your environment by adding
  commands to the end of your .bashrc file , such
  as alias commands. (Again do this with care! )

22
Some basic rules

• Unix is case sensitive.
• Commands are in lower case.
• Backspace and/or Del Keys correct typing errors.
  If the terminal parameters are not correctly set
  try CtrlH
• CtrlC Aborts a program or command.
• You can use the arrow keys to recall previous
  commands, optionally edit and execute them.

23
Key combinations in bash
CtrlA Move cursor to beginning of line
CtrlC ltBreakgt Stop running program
CtrlD ltLogoutgt
CtrlE Move cursor to end of line
CtrlH Delete character to the left
CtrlZ Suspend the program
Arrow left/right Move pointer left/right to insert chars etc.
Arrow up/down Recall previous commands so as to edit them andsubmit them by pressing ENTER
Tab Tab tab Command or filename completion. Auto-complete the command line (to save typing) If tab does not work tab again to show possibilities
24
Format of Unix commands

• command option ... filename ...
• eg ls
• ls -l tutorial
• more tutorial

25
List Directory

• ls list directory
• ls -l list directory in long format
• ls -a list all (inc. hidden) files
• -rw------ l course01 57 Oct 18 1105
  hello.c

Access Permissions
Number ofbytes in file
Date and time last modified
26
Directory Structure
/ (root)
home
usr
cs4un2
cs4un1
Home directory of user cs4un1 /home/cs4un1
When you log in you are positioned in your home
directory. The environment variable HOME is also
set to contain this directory name.
27
Working with Directories

• pwd print working directory
• cd change directory
• cd move to home directory
• cd .. move up one level
• cd mydir move into a subdirectory
• cd /var/adm move to an absolute directory
• mkdir directory_name create a new directory
• rmdir directory_name delete an empty directory

28
Filenames

• Filenames can comprise of
• a-z, A-Z alphabetic characters
• 0-9 digits
• .-_ special characters
• montue_01.06-03-96
• Wildcards when referencing files
• any character or sequence of characters
• ? any single character

29
Displaying contents of a text_file

• more filename
• This command will start listing the contents of
  filename on screen and pause after a screenfull
  of data. While pausing, use the following
  characters to control the output.
• Spacebar next screenful
• n Spacebar next n lines
• Enter next line
• b back one screen
• n b back n screens full
• q quit
• ? or h list commands
• where n is a whole number

30
Displaying contents of a text_file continued

• cat options filename filename
• This command will output the contents of
  filenames to standard-output ( normally screen)
  without pausing.
• Following options are useful
• -v display non-printing characters
• -n display with lines numbered on the left
• tail -n filename
• This command lists the last 10 lines of a text
  file.
• If a number is specified (.eg. -20 ) lists the
  last n (i.e 20) lines

31
Copying files

• Copy files (optionally directories)
• cp fromfile tofile
• Some of the useful options are
• -R or r Recursive copy fromfile is a
  directory so the entire directory and its
  contents are copied.
• e.g. cp r mydir newdir
• -p preserve. Preserves all attributes of
  the file ,such as access rights and creation
  date.
• Copy and concatenate files by using cat
• Cat command concatenates contents of list of
  files and directs the output to standard output
  (normally screen). When used with redirection gt
  it can be used to join files together.
• e.g. cat file1 file2 file3 gt new_big_file

32
Renaming and deleting files

• mv This command will move a file or directory to
  a new location. It can thus be used to rename
  files/directories as well as change their
  locations in the global directory structure.
• Syntax
• mv source destination
• Example
• mv myfile mynewfile
• mv myfile subdirectory/myfile
• mv mysubdir mynewsubdir
• rm This command will delete a file (optionally
  a directory if used with r option).
• Syntax rm object_to_delete
• Example
• rm myfile
• rm r mydirectory

33
Working with files

• To copy a file cp my_file my_new_file
• To move ( or rename ) a file mv my_file
  my_new_file
• To delete a file rm my_file
• To list the contents of a file less
  file_name
• To make a new directory( i.e. folder) mkdir
  new_directory
• To copy a file into another directory cp my_file
  other_directory
• To move a file into another directory mv my_file
  other_directory
• To copy a directory to another location cp R
  directory other_directory
• To remove a directory and all its contents!!! rm
  R directory
• ( use with care )
• Wildcards means matching any sequence of
  characters.
• For example cp .dat my_directory

34
Problem Session

• Attempt problem 1-5 on the handout

35
Transferring files to/from iceberg

• Summary of file transfer methods as well as links
  to downloadable tools for file transfers to
  iceberg are published at http//www.sheffield.ac
  .uk/wrgrid/using/access
• Command line tools such as scp, sftp and gftp are
  available on most platforms.
• Can not use ftp ( non-secure ) to iceberg.
• Graphical tools that transfer files by dragging
  and dropping files between windows are available
  winscp, coreftp, filezilla, cyberduck

36
Pitfalls when transferring files

• ftp is not allowed to by iceberg. Only sftp is
  accepted.
• Do not use spaces in filenames. Linux do
  not like it.
• Secure file transfer programs sftp classify
  all files to be transferred as either ASCII_TEXT
  or BINARY.
• All SFTP clients attempt to detect the type of a
  file automatically before a transfer starts but
  also provide advanced options to manually declare
  the type of the file to be transferred.
• Wrong classification can cause problems
  particularly when transfers take place between
  different operating systems such as between Linux
  and Windows.
• If you are transferring ASCII_TEXT files to/from
  windows/Linux, to check that transfers worked
  correctly while on iceberg, type cat v
  journal_fileIf you see a M at the end of each
  line you are in trouble !!!
• CURE dos2unix wrong_file on iceberg

37
Your filestore

• Three areas of filestore always available on
  iceberg
• These areas are available from all headnode and
  worker nodes.
• home directory
• /home/username
• 5 GBytes allocations
• Permanent, secure, backed up area ( deleted files
  can be recovered )
• data directory
• /data/username
• 50 GBytes of ollocation
• Not backed but mirrored on another server
• /fastdata area
• /fastdata
• Much faster access from MPI jobs
• No storage limits but no backup, or mirroring
• Files older than 90 days gets deleted
  automatically
• Always make a directory under /fastdata and work
  there.

38
Scratch area ( only available during a job)

• Located at /scratch
• Used as temporary data storage on the current
  compute node alone.
• File I/O to /scratch is faster than to NFS
  mounted /home and /data areas
• File I/O to small files is faster than to
  /fastdata but
• for large files /fastdata is faster than
  /scratch
• Data not visible to other worker nodes and
  expected to exist only during the duration of the
  job.
• HOW TO USE SCRATCH
• Create a directory using your username under
  /scratch on a worker and work from that directory
• Example mkdir /scratch/USER
• cp mydata /scratch/USER
• cd /scratch/USER
• ( then run your program )

39
Storage allocations

• Storage allocations for each area are as follows
• On /home 5 GBytes
• On /data 50 GBytes
• No limits on /fastdata
• Check your usage and allocation often to avoid
  exceeding the quota by typing quota
• If you exceed your quota, you get frozen and the
  only way out of it is by reducing your
  filestorage usage by deleting unwanted files via
  the RM command ( note this is in CAPITALS ).
• Requesting more storage
• Email iceberg-admins_at_sheffield.ac.uk to request
  for more storage.
• Excepting the /scratch areas on worker nodes, the
  view of the filestore is the same on every worker.

40
Running tasks on iceberg

• Two iceberg headnodes are gateways to the cluster
  of worker nodes.
• Headnodes main purpose is to allow access to the
  worker nodes but NOT to run cpu intensive
  programs.
• All cpu intensive computations must be performed
  on the worker nodes. This is achieved by
• qsh command for interactive jobs and
• qsub command for batch jobs.
• Once you log into iceberg, taking advantage of
  the power of a worker-node for interactive work
  is done simply by typing qsh and working in the
  new shell window that is opened. The next set of
  slides assume that you are already working on one
  of the worker nodes (qsh session).

41
Where on the cluster ?

• Most of the application packages, compilers and
  software libraries are only available on the
  worker_nodes.
• Iceberg headnodes are suitable for only
  light-weight jobs such as editing files.
• If you are on one of the headnodes, you will need
  to type qsh to get an interactive session to the
  worker nodes.
• How do you know where you are ? The command
  prompt will contain your userid_at_hostname
  Example ch1abc_at_node-056
• If you are on one of the headnodes, hostname will
  be iceberg1 or iceberg2
• If you are on an amd-based worker-node, it will
  be amd-nodenn
• where nn is the number of the amd-node.
• If you are on an intel-based worker-node it will
  be node-nnn
• where nnn is the number of the intel node.
• You can always type echo HOSTNAME to find out
  the name of the machine you are currently using.

42
Running programs on iceberg

• Iceberg is the gateway to the cluster of worker
  nodes and the only one where direct logging in is
  allowed.
• Icebergs main purpose is to allow access to the
  worker nodes but NOT to run cpu intensive
  programs.
• All cpu intensive computations must be performed
  on the worker nodes. This is achieved by the qsh
  command for the interactive jobs and qsub command
  for the batch jobs.
• Once you log into iceberg, taking advantage of
  the power of a worker node for interactive work
  is done simply by typing qsh and working in the
  new shell window that is opened. This what
  appears to be a trivial task has would in fact
  have queried all the worker nodes for you and
  started a session on the least loaded worker in
  the cluster.
• The next set of slides assume that you are
  already working on one of the worker nodes (qsh
  session).

43
Running programs

• Two modes of operation foreground and background
• Foreground Interact with program via
  keyboard/screen
• Background No connection with keyboard/screen
• Submit to backbround by Appending
• EG myprog gt myfile
• The symbols gt redirect output and any errors
  to the file myfile
• Although the above method of running jobs on the
  background is feasible, the prio we recommend
  that you submit your background into the batch
  queue via the qsub command.

44
Redirection

• Most unix commands are not aware of the source of
  their input or the destination of their output.
  They simply read/write from/to stdin/stdout. The
  shell takes care of these issues.
• Standard Input (defaultgtkeyboard)
• Standard Output (defaultgtscreen)
• Redirection symbols lt,gt,gtgt can be used to specify
  files as the source/destination of the read/write
  operations to override the above defaults.

45
Redirection continued

• Most unix commands are not aware of the source of
  their input or the destination of their output.
  They simply read/write from/to stdin/stdout. The
  shell takes care of these issues.
• To redirect the output to a file use the gt
  symbol.
• Example ls -l gt dirlist
• The gt symbol should be used with care as it
  may over-write an existing file. gtgt symbol can
  be used instead if the output should be appended
  to the end of an existing file rather than
  over-writing it.
• Example ls -l gtgt logfile
• If nothing is directed to a file then a zero size
  file is created, or if the file already existed
  then the contents of the file is removed
• Example gt afile
• The file /dev/null is a special symbol to
  indicate a black-hole
• Example ls l gt /dev/null

46
Redirection continued

• If any program expects any of its input from the
  standard-input-channel , i.e. the keyboard, it
  can also read the same information from a file by
  redirection.
• To read input from a file use the lt symbol.
• Example write cs1xyz lt message.fil
• Here any text input write program expects from
  the keyboard will be simply read from a file
  named message.fil.
• Each end-of-line will be treated as an ltENTERgt
  on the keyboard.

47
Piping

• Feeding the output of one command into the input
  of another command
• The symbol is called a pipe
• command command
• eg ls -al more
• ls -la grep Nov

48
Examples of re-direction and piping

• ls l grep Jun
• ls l grep Jun gt june_files
• ls l grep Jun cut c 57-80 gt june_files
• cut c 1-10 lt test_files
• aspell l lt message.txt gt report.txt
• grep fluent lt news.dat
• grep fluent lt news.dat cut c 1-72
• (grep fluent lt news.dat ) cut c 1-72
• (grep fluent lt news.dat ) cut c 1-72 gt
  fluent.news

49
Foreground Program Control

• Kill a program Ctrl C
• Stop a program Ctrl Z
• Note a stopped program still exists in the system
• and hence can be re-started.

50
Program control within current shell

• jobs Lists jobs (programs)
• bg job_id Place a job in the background
• fg job_id Return a job to the foreground
• stop job_id Stop a job
• kill job_id Kill a job
• Process_id can be used in place of job_id for
  more definitive way of identification.
• jobs
• 1 Running time.sh gt out
• stop 1
• 1 Stopped (signal) time.sh gt out
• bg 1
• 1 time.sh gt out
• kill 1
• Terminated

51
Program control using ps and kill

• ps Report process status
• ps -f -u username
• UID PID PPID C STIME TTY TIME CMD
• username 24816 24585 0 162304 pts/50 000
  sleep 2
• username 20169 19956 0 160545 pts/50 001
  -csh
• username 24585 20169 0 163507 pts/50 000
  /bin/sh time.sh
• kill Terminate process
• Find its process ID (PID) using the ps command
• Kill the process using the kill command
• kill 24585
• Sometimes kill on its own does not work so try
• kill -KILL 24585 or kill -9 pid

52
Repeating Previous Commands

• Operation
• history List previous commands
• !! re-run last command
• !n re-run the nth command
• !str last command starting with str
• eg !vi
• Setup
• Add the following to your .cshrc file
• set history40

53
Searching in files

• grep string file
• This command finds and prints out the lines in
  the file(s) containing the specified string
• string word or phrase
• file file or list of files (wild_card can be
  used)
• Note We strongly advise that the string is
  quoted.
• Examples grep Green Man england.dat
• grep Zodiac t.dat
• grep Zonea-z security.fil

54
Manual Pages and Info system

• Man Manual pages Give text-based help on
  usage.
• Manual pages are grouped into sections (1,2,3,4
  ..).
• There is usually one manual page per command
  which is located in one of the directories
  defined by the MANPATH environment variable.
• To access the Manual page for a command just
  type
• man command
• To get a list of manual pages that contain a
  word type
• man k topic
• Info Similar to man but can scroll with cursor
  keys and link to other information.

55
Further Help

• Getting started with iceberg
• http//www.shef.ac.uk/wrgrid/iceberg
• http//www.shef.ac.uk/wrgrid/using/introlinux
• http//www.shef.ac.uk/wrgrid/using/introbash
• GNU/Linux Command-Line Tools Summary
• http//en.tldp.org/LDP/GNU-Linux-Tools-Summary/htm
  l/index.html
• Bash Guide for Beginners
• http//tldp.org/LDP/Bash-Beginners-Guide/html/inde
  x.html
• Advanced Bash-Scripting Guide
• http//en.tldp.org/LDP/abs/html/index.html
• The Linux Documentation Project
• http//en.tldp.org/

56
Problem Session

• Attempt problem 6-9 on the handout