Chapter 3 UNIX Utilities II

1
Chapter 3 UNIX Utilities II

• By C. Shing
• ITEC Dept
• Radford University

2
Objectives

• Understand how to filter files
• Understand how to sort files
• Understand how to search for files
• Understand how to compare files
• Understand how to archive, transform and
  compress/uncompress files
• Understand how to schedule processes
• Understand hard and soft links
• Understand how to attach different file systems
• Understand how to find every character in file

3
Filters

• grep (Global or Get Regular Expression and
  Print), egrep (Extended grep), fgrep (Fast grep)
• uniq
• awk (Aho, Weinberger, Kernighan) C control
  structures
• sed (Stream Editor)

4
Filters

• grep, egrep, fgrep filter out lines that
  do/dont have specified pattern in the file
  content
• grep
• Syntax grep -option pattern filename
• Option
• i ignore case
• l list files containing the pattern
• n display line
• v displays lines that don't match the pattern
• w match to whole word only
• Pattern regular expression

5
Filters- grep
6
Filters- grep
7
Filters- grep

• Examples
• grep smith /etc/passwd
• show all lines that have smith pattern
• grep -wn lower /usr/include/
• use word lower and show line
• grep -ni lower /usr/include/
• ignore case and show line number
• ls -l grep d
• show directory
• ls -l grep ..w
• show files that owner can write
• cut -d -f5 /etc/passwd grep -i ' smith'
  more
• show under field 5, any line has word end with
  smith case insensitive

8
Filters- egrep

• egrep
• Syntax egrep -option pattern filename
• or grep e -option pattern filename
• Option
• i ignore case
• l list files containing the pattern
• n display line
• v displays lines that don't match the pattern
• w match to whole word only
• Pattern extended regular expression

9
Filters- egrep
10
Filters- egrep

• Examples
• egrep sm.?ith /etc/passwd
• all lines contain 0 or 1 of any character after
  sm
• egrep -wn stdio \ STDIO /usr/include/stdio.h
• any line containing stdio or STDIO
• egrep -ni to /usr/include/ grep too
• all lines contain too
• ls -l /devices egrep v d
• show all non-directory files
• (device has major routine , minor index
  in the routine)
• cut -d -f5 /etc/passwd egrep -i ' smith.'
  more
• any line containing smith followed by 1 or more
  characters

11
Filters- fgrep

• fgrep
• Syntax fgrep -option string filename
• Option
• i ignore case
• l list files containing the pattern
• n display line
• v displays lines that don't match the pattern
• w match to whole word only

12
Filters- fgrep

• Examples
• fgrep smith /etc/passwd
• fgrep wn lower /usr/include/

13
Filters- uniq

• uniq filter out duplicated adjacent lines
• Syntax uniq -option infile outfile
• Option
• number of fields to be skipped (delimiter
  space)
• c precedes of duplications

14
Filters- uniq

• Examples
• cut d -f4 /etc/passwd uniq -c
• list of same adjacent GID lines

15
Compare - cmp

• cmp find 1st byte that is different
• Syntax cmp -option file1 file2
• Option
• l list offset and line number of 1st mismatched
  byte
• S all output inhibited

16
Compare - diff

• Diff display differences so file1 can be
  converted to file2
• Syntax diff -option file1 file2
• Option
• i ignore case

17
Compare - diff

• Output form
• File1Line a File2LineStart, File2LineStop
• File1LineStart, File1LineStop d lineCount
• File1LineStart, File1LineStop c
  File2LineStart, File2LineStop
• ---

18
Compare - diff

• Example
• diff /etc/auto_net_rucs /etc/auto_net_rucs2
• diff /etc/auto_net_rucs /etc/auto_net_ruacad

19
Sort

• Sort in ascending order
• Syntax sort option file
• Option
• t c-delimiter
• default - space or tab
• n1 n1 is the index number for the starting sort
  field
• -n2 n2 is the stopping sort field index number
  (non-inclusive)
• b ignore leading blanks
• f ignore case for sort fields
• n sort fields in numerical order
• M sort fields in month order
• r sort in reverse (descending) order
• n3, -n4 more comparison when equal in using
  n1,-n2

20
Sort

• Examples
• sort 0 -1 -b /usr/include/stdio.hstarts from
  and to the 1st field (index0), ignore leading
  spaces and sort /usr/include/stdio.h in ascending
  order (delimiter is space or tab).
• sort -t 4 -5 -b 0r /etc/passwd moreUse as
  the delimiter of fields, sort in ascending order
  according the Real Name (index4) in /etc/passwd,
  ignore leading spaces if sorted fields are
  equal, then sort by the username in descending
  order
• sort -t 2 -3n /etc/passwd moresort by uid
  (numerically)

21
Search

• Find search the expression from the path
• Syntax find path expression

22
Search
23
Search
24
Search
25
Search

• Examples
• find -name '.c' -print moreIt recursively
  goes down from your home directory to find any c
  files and print on screen.
• find . -mtime -14 print more
• It displays all files that has been modified
  within the past 2 weeks.

26
Search

• Examples
• (find / -name handler.py /dev/tty)
  /dev/null
• This finds the python file and sends errors to
  /dev/null, and results are displayed on the
  terminal, so you dont see error in screen
• find / -name handler.py errorfile
• The error will goes to the file errorfile

27
Search (Cont.)

• Examples (Cont.)
• find . -name a.out -ls -exec rm \It
  recursively goes down from the current directory
  to delete a.out files and print those names on
  screen. (This rm command may execute 100 times if
  there are 100 a.out files found)
• Note a more efficient way is to use xargs
  command as below (Linux only)
• find . -name a.out -ls xargs rm
• (This will put all files found in the
  parameters of the command rm and execute once)
• If the command line parameters of rm has limited
  to 10, then we can use
• find . -name a.out -ls xargs n10 rm
• (This will execute the command 10 times, each
  time it removes 10 files.)

28
Search (Cont.)

• Examples (Cont.)
• find . \( -name '.c' -o -name '.java' \) -print
  moreIt recursively goes down from the current
  directory to find any c files or java files and
  print on screen.

29
Filters - awk

• awk scan every line that matches certain
  criteria (Programmable Text Processing ) or no
  condition, perform some actions to them
• Use C syntax and C library function
• Syntax each line in form - conditionaction
• awkPgm (use f) line in file
• awk -option -f awkPgm file
• Command line (in quotes)
• awk -option 'conditionaction' file
• Option
• Fc c Fields delimiter
• tab or space - default delimiter

30
Filters - awk

• Condition

31
Filters - awk

• Action

32
Filters - awk

• Action

33
Filters - awk

• Examples(all necessary Unix format files(Do not
  use DOS format) are in the folder)
• awk -F 'print NF, 0' /etc/passwdmorewhere
  NF of fields (predefined variable),
• 0 entire line
• awk -F -f awk1 /etc/passwdmoreIt prints 1st,
  3rd fields and number of fields (no spaces in
  between)

34
Filters - awk

• awk -F -f awk2 /etc/passwd moreIt prints out
  1st, 3rd fields and number of fields (with spaces
  in between) for line 2 and 3(NR record number)
• awk -F -f awk3 /etc/passwd moreIt prints out
  line along with each line and count total of
  line s and fields.
• awk -F -f awk4 /etc/passwd moreIt prints out
  from the last field back to the 1st field for
  each line.

35
Filters - awk

• awk -F -f awk5 /etc/passwd moreIt prints out
  the line that have the pattern (t followed by one
  or more characters and end with e).
• awk -F -f awk6 /etc/passwd moreIt prints out
  the line that contains "smith".
• awk -F -f awk7 /etc/passwd moreIt prints out
  the line that contains "Smith".

36
Filters - awk

• awk -F -f awk8 /etc/passwd moreIt prints out
  all lines in between (inclusively) the beginning
  line that contains "root" to the ending line that
  contains "local
• awk -f awk9 awktestlibIt shows awk library
  functions.

37
Filters - awk

• 11. Example of using if .. else

38
Filters - sed

• sed scan every line and perform simple editing
  action on lines that match certain criteria
• Syntax sed -option scriptfile filename
• Option
• f scriptfile is file of command lines
• no option use command line in quotes
• n suppress print duplication caused by p
• e multiple instructions in command line

39
Filters - sed
40
Filters - sed
41
Filters - sed

• Examples(all necessary Unix format files(Do not
  use DOS format) are in the folder)
• sed 's//\ /' awk1 awk1.indentThis gives a
  at the beginning of each line in the file awk1
  and creates output file awk1.indent.
• sed 's/ //' awk3This removes leading spaces in
  each line of the file awk3.
• sed -f sed1 awk3This removes all lines that
  contains in the file awk3.
• sed '2,4d' awk9This removes line 2 to line 4 in
  the file awk9.

42
Filters - sed

• sed 'd' awk9This removes last line in the file
  awk9.
• sed -n '2,4p' awk9This prints from line 2 to
  line 4 in the file awk9.
• sed 5q awk9This prints from line 1 to line 5 in
  the file awk9.
• sed f sedinsert awk9
• This inserts 1st and last line.
• sed f sedchange awk9
• This changes the last line.
• sed f sedreplace awk9
• This replace all printf by fprintf
• sed 1,s/the/THE/g awk9
• replace the word the by THE in the entire
  file awk9
• sed n e 1,2p e p awk9
• prints line 1,2 and the last line of the file
  awk9
• sed n e //,//w table.html e
  //,//w frame.html entire.html
• extract table and frame parts of the entire.html
  and save them into individual files

43
Archive

• cpio (not in MS-DOS) small files
• tar (also available in MS-DOS) backup on tape
  - single volume
• dump (or ufsdump) incremental backup can store
  in many volumes
• restore (or ufsstore) recover

44
Archive - cpio

• cpio
• Syntax cpio option filedirectory
• Option
• o takes standard input and creates output cpio
  format files with those filenames
• v displays those filenames
• i takes a cpio format file from standard input
  and creates those filenames
• d creates directory
• t creates table of contents
• p copies files into the directory
• l creates links (instead of copying) in the
  directory

45
Archive - cpio

• Examples
• mkdir tmpcp .c tmpcd tmpls .c cpio -ov
  backupIt creates a cpio format output file
  conatining all c files
• ls -l backuprm -f .clscpio -i recreates all original files from the cpio
  file.ls -lcd ..rm -rf tmp

46
Archive - tar

• tar
• Syntax tar option tarfilename filelist
• Option
• c creates tar format file
• f tarfile
• r unconditionally append to the tarfile
• t creates table of contents
• v verbose output
• x extract from tarfile
• Z compress file

47
Archive - tar

• Examples
• tar -cvf tarfile .ctar all c files to the
  current firectory
• tar -tvf tarfileread current contents of tarfile
• tar -rvf tarfile .txtunconditionally adds all
  txt files to the end of the tarfile
• tar -tvf tarfile moremkdir tmpmv tarfile
  tmpcd tmp
• tar -xvf tarfile tar -tf tarfile grep filecp.c
  It only untar the file filecp.c from the
  tarfile.
• Tar czf tarcompressfile
• achive and then compress file

48
Archive dump/restore

• dump
• Syntax dump level f option dumpfile
  filesystemwhere dumpfile is defaulted to
  /etc/rmt0
• Option
• v verify each volumn
• w displays filesystems that need to be dumped

49
Archive dump/restore

• Examples
• dump 0 fv /dev/rmt0 /dev/da0

50
Archive dump/restore

• restore
• Syntax restore -option f dumpfile files where
  dumpfile is defaulted to /etc/rmt0
• Option
• t show table of contents
• x restore only files specified
• i interactive mode

51
Archive - dump/restore

• Examples
• restore -x f /dev/rmt0 file1.c file2.c

52
Transform

• Tr translate from string1character set to
  string2 character set, respectively. If
  len(string2)
  in string2 as long as possible
• Syntax tr option string1 string2
• Option
• c complement of string1 set
• d string1 set deleted
• s show once for duplicated characters

53
Tee

• Duplicates input and save it in a file and
  creates standard output
• Example
• who tee who.txt wc l
• This saves users in the file who.txt and output
  number of users in screen

54
Transform

• Example
• tr lower upper more
• change from lower case to upper case
• tr a-d A-D
• tr cs a-z \012
• for non-(a-z) character, replace by new-line
  character only once

55
Compress/Uncompress

• Compress/Uncompress (System V)
• Syntax
• compress -option file
• uncompress file.Z
• Option
• c send result to screen, instead of replacing
  the old file by a .Z file
• v verbose output

56
Compress/Uncompress

• Examples
• compress v filename
• uncompress v filename.Z

57
Compress/Uncompress

• gzip/gunzip
• Syntax
• gzip -option file
• gunzip file.gz
• Option
• c send result to screen, instead of replacing
  the old file by a .gz file
• f force
• GNU free compress program

58
Compress/Uncompress

• zip/unzip (Linux)
• Syntax
• zip -option file.zip file1 file2
• unzip file.zip
• Option
• r recursively descend down its tree to the
  leaves
• PKZIP/PKUNZIP compress program

59
Schedule Process

• Periodic execution crontab
• only users in cron.allow can use if the file
  exists
• all users in cron.deny are not allowed to use if
  the file exists
• One time execution at
• similar for files at.allow and at.deny
• One time execution when CPU is available batch
• e.g. batch

60
Schedule Process - crontab

• crontab schedule a crontab process
  periodically
• Syntax crontab -option
• Option
• l lists contents of a registered crontab file
• e update and reregister the registered crontab
  file
• r unregister the registered crontab file

61
Schedule Process - crontab

• A crontab file must be in the Unix form (not
  DOS)
• minute hour day month weelday command See the
  file crontab.cron in the folder
• Note
• Avoid to use for minute field
• All fields
• List e.g. 2,4,6
• Range e.g. 2-6/2 (as above)
• 2-6

62
Schedule Process - crontab

• Examples
• crontab crontab.cronThis registers the crontab
  file
• crontab -eThis can edit the crontab file and
  reregister it
• crontab -lThis lists the contents of the crontab
  file
• crontab -rThis unregister the crontab file.

63
Schedule Process - at

• at schedule a one-time process
• Syntax
• at time shellscript at the future time.
• at -l
• This checks all future jobID.
• at -r jobID
• This removes the future jobID.

64
Schedule Process - at

• Examples (See the file at.csh in the folder)
• at 430pm executed at 430pm today.
• at now 1 day be executed at exactly one day from now.
• at 5pmThis sets interactive mode to type in all
  shell command and finish it be ctrl-d

65
Unix File System

• Introduction

66
Link - hard

• Hard link
• Syntax ln filename aliasename
• Hard link will creates an alias called aliasname
  having the same inode number (same file)

67
Link hard (Cont.)

• Limitations
• Cant link file across different file systems
• Cant link directory

68
Link - hard

• Examples link directory
• mkdir temp
• ln .c tempThis creates hard links to all c
  files and put them in temp directory
• ls li
• cd temp
• ls -li

69
Link - soft

• Soft link
• Syntax ln -option filenamedirectory softlink
• Option
• s soft link
• Soft link creates a different file
  (different inode), it can link to different file
  systems
• Note This is similar to short-cut in Windows.

70
Link - soft

• Example usually used to link directory
• ln -s /usr/include myincludedir
• cd myincludedir
• ls -l

71
Link - soft

• Example link file
• ln file1 file2
• ln s file1 file3
• cat file2
• cat file3
• ls i file
• rm f file2
• ls i file
• rm f file1
• ls i file
• ls l file
• cat file3
• file3 becomes a dangling link
• Check their inode and file content in file3

72
Attach/Detach File System mount/umount

• Mount/umount attach/detach file system
• Syntax
• mount -o options devicename mountpoint
• Option
• ro read only
• rw read and write (default)
• mount
• list currently mounted devices
• umount mountpoint
• detach file system

73
Attach/Detach File System mount/umount

• Examples Use jump drive on Red Hat Linux
• mkdir /usb
• mount /dev/sd0 /usb
• mount
• cp /usb
• umount /usb

74
Raw File Content - od

• od dump non-text file content in octal form
• Syntax od option filename
• Option
• c in character form
• sn search string of length n (default 3)
• x in hexadecimal form
• b in Octal form (Linux)

75
Raw File Content - od

• Examples
• od c java1.class more
• look at characters in bytecode file java1.class
• od cxs5 a.out more
• look at characters (string length at least 5,
  also in hexadecimal form) in executable file
  a.out
• od bc a.out more

76
Misc.

• whoami
• cut
• ul transform an underline sequence file to an
  ASCII text file
• man cut ul tdumb cut.txt
• spell spell checker
• crypt keyvalue cryptFileTo encrypt
  the file origFile.
• crypt keyvalue cryptFile.

77
Misc.

• time commandFind real time, user and system CPU
  times for running the command.
• paste file1 file2
• Align 2 files side by side
• nohup command
• Do not kill process created by the command when
  logout

78
Misc.

• dfcheck usage of mounted file systems
• dos2unix dosfilename unixfilename
• convert dos file to unix file
• unix2dos unixfilename dosfilename
• convert unixfile to dosfile
• last username show last login time
• strings executable examine executable string
  contents
• dd create bit by bit image copy
• paste file1 file2 put file2 side by side with
  file 1

79
Perl

• Introduction

80
Reference

• Appendix A