Advanced UNIX

1
Advanced UNIX
240-491 Special Topics in Comp. Eng. 1Semester
2, 2000-2001
4. Filters(Part II, Sobell)

• Objectives
• to discuss five useful filterstr, grep, awk,
  sed, and find

2
1. tr

• format
• tr options string1 string2
• tr reads its standard input and translates each
  character in string1 to the corresponding
  character in string2

3
Examples

• echo 12abc3def4 tr abcdef
  xyzabc12xyz3abc4
• echo 12abc3de4 tr a-cd-f
  x-za-c12xyz3abc4
• cat foo.txt tr A-Z a-z

4

• tr \015 file2
• \015 is carriage return
• cat mail.txt tr -s ?
  new-mail.txt
• ? represents tab could write \011
• -s means remove duplicates of string2 in output
• echo Can you read this? tr -d
  aeiouCn y rd ths?

5
rot13 Text
Popular in 1970-1980s.

• echo Gur chapuyvar bs gur wbxr vf ... tr
  N-ZA-Mn-za-m A-MN-Za-mn-z
  The punchline of the joke is ...

6
2. grep

• Format
• grep options pattern file-list
• Search one or more files, line by line, for a
  pattern (a regular expression). Actions taken
  depend on options.

7
Variants of grep

• grep Uses basic RE pattern
• fgrep Fast grep. Pattern can only be an
  ordinary string.
• egrep Extended grep. Pattern can use full REs.

8
grep options

• -c print a count of matching lines
• -i ignore case in pattern during search
• -l list filenames with match
• -n precede each matching line by a line
  number
• -v print lines that do not match pattern

9
Examples

• File testa File testb File testcaaabb aaaaa
  AAAAAbbbcc bbbbb BBBBBff-ff ccccc CCCCC
  cccdd ddddd DDDDDdddaa

continued
10

• grep bb testaaaabbbbbcc
• grep -v bb testaff-ffcccdddddaa
• grep -n bb testa1 aaabb2 bbbcc

continued
11

• grep bb testa aaabbtesta bbbcctestb
  bbbbb
• grep -i bb grep -i BB testa
  aaabb testa aaabb testa bbbcc testa bbbcc
  testb bbbbb testb bbbbb testc
  BBBBB testc BBBBB

12
Fancier Patterns

• grep fun..ion file
• grep -n define file
• grep dea-z file
• egrep whileif .c
• egrep 0-9 .c

13
3. awk

• format
• awk program file-list
• awk -f program-file file-list
• awk is a pattern scanning and action processing
  language
• The action language is very like C.

14
Overview

• 3.1. Patterns Actions
• 3.2. awk Processing Cycle
• 3.3. How awk Sees a Line
• 3.4. Pattern Expressions
• 3.5. , Range Operator

continued
15

• 3.6. Many Built-in Functions
• 3.7. BEGIN and END
• 3.8. First awk Program File pre_header
• 3.9. Action Language
• 3.10. Associative Arrays

16
3.1. Patterns Actions

• An awk program consists of
• pattern actionpattern action

17
3.2. awk Processing Cycle

• 1. Read next input line.
• 2. Apply all awk patterns sequentially.
• 3. If a pattern matches, do its action.
• 4. Go to step (1).

18
Example

• cat carsplym fury 77 73 2500chevy nova 79 60 3
  000ford mustang 65 45 10000volvo gl 78 102 9850
  ford ltd 83 15 10500chevy nova 80 50 3500fiat 60
  0 65 115 450honda accord 81 30 6000ford thundbd
  84 10 17000toyota tercel 82 180 750chevy impala
  65 85 1550ford bronco 83 25 9500

continued
19

• awk /chevy/ print carschevy nova 79 60 3000
  chevy nova 80 50 3500chevy impala 65 85 1550
• awk /chevy/ carschevy nova 79 60 3000chevy n
  ova 80 50 3500chevy impala 65 85 1550
• awk /h/ carshonda accord 81 30 6000

20
3.3. How awk Sees a Line

• awk views each line as a record consisting of
  fields separated by spaces.
• Each field is referred to by a variable called
  
• 1, 2, 3, etc.
• 0 refers to the whole line (record)
• The current line number is stored in NR

continued
21

• awk print 3, 1 cars77 plym79 chevy65
  ford 83 ford
• awk /chevy/ print 3, 1 cars79 chevy80
  chevy65 chevy

22
3.4. Pattern Expressions

• Format
• variable OP pattern
• OP forms
• matching !
• ariithmetic
• boolean !

continued
23

• awk 1 /h/ carschevy nova 79 60 3000chevy
  nova 80 50 3500honda accord 81 30 6000chevy im
  pala 65 85 1550
• awk 1 /h/ carshonda accord 81 30 6000

continued
24

• awk 2 /tm/ print 3, 2, 5
  cars65 mustang 1000084 thundbd 1700082
  tercel 750
• awk 3 /5/ print 3, 1, 5
  cars65 ford 1000065 fiat 45065 chevy 1550

continued
25

• awk 3 65 carsford mustang 65 45 10000fia
  t 600 65 115 450chevy impala 65 85 1550
• awk 5 y nova 79 60 3000fiat 600 65 115 450toyota ter
  cel 82 180 750chevy impala 65 85 1550

continued
26

• awk 5 2000 5 carsplym fury 77 73 2500chevy nova 79 60 3000
  chevy nova 80 50 3500fiat 600 65 115 450honda
  accord 81 30 6000toyota tercel 82 180 750
• awk 5 2000 5 carsplym fury 77 73 2500chevy nova 79 60 3000
  chevy nova 80 50 3500honda accord 81 30 6000

27
3.5. , Range Operator

• Format
• pattern1 , pattern2
• Select a range of lines.
• the first line of the range matches pattern1
• the last line of the range matches pattern2
• May return several groups of lines

continued
28

• awk /volvo/ , /fiat/ carsvolvo gl 78 102 985
  0ford ltd 83 15 10500chevy nova 80 50 3500fia
  t 600 65 115 450
• awk NR 2 , NR 4 carschevy nova 79 60 30
  00ford mustang 65 45 10000volvo gl 78 102 9850

continued
29

• awk /chevy/ , /ford/ carschevy nova 79 60 30
  00ford mustang 65 45 10000chevy nova 80 50 3500
  fiat 600 65 115 450honda accord 81 30 6000ford
  thundbd 84 10 17000chevy impala 65 85 1550ford
  bronco 83 25 9500

threegroups
30
3.6. Many Built-in Functions

• length(str) length of string strlength length
  of current line
• split(strings, array, delimitor) split string
  into parts based on the delimitor, and place
  in array
• split(a bcd ef g1, arr, )

continued
31

• awk length 23 print NR cars3910

32
3.7. BEGIN and END

• BEGIN action executed before first line is
  processed
• END action executed after last line is
  processed

• awk END print NR, cars for sale.
  cars12 cars for sale

33
3.8. First awk Program File

• cat pr_header pr_headerBEGIN print
  Make Model Year Miles Priceprint
  ---------------------------------
  print

continued
34

• awk -f pr_header carsMake Model Year
  Miles Price---------------------------------
  plym fury 77 73 2500chevy nova 79
  60 3000 chevy impala 65
  85 1550ford bronco 83 25 9500

35
redirect_out

• cat redirect_out/chevy/ print
  chev.txt/ford/ print ford.txtEND
  print done.
• awk -f redirect_out carsdone. cat
  chev.txtchevy nova 79 60 3000chevy nova 80 50
  3500chevy impala 65 85 1550

36
3.9. Action Language

• Very C like
• var expr
• if (cond) stat1 else stat2
• while (cond) stat
• for (expr1 cond expr2) stat
• printf format expr1, expr2, ...
• stat1 stat2 ... statN
• User-defined variables do not need to be declared

continued
37

• Long statements, conditions, expressions may need
  to be typed over several lines.
• Use \ to hide newline
• if (3 2000 \ 3

38
price_range

• cat price_rangeif (5 inexpensiveelse if (5 5000 5 \ 5 please askelse if (5 10000) 5
  expensiveprintf -10s -8s 192d 5d
  -12s\n, \ 1, 2, 3, 4, 5

continued
39

• awk -f price_range carsplym fury 1977 73 inexp
  ensivechevy nova 1979 60 inexpensive ford
  bronco 1983 25 please ask

40
summary

• cat summaryBEGIN yearsum 0 costsum 0
  newcostsum 0 newcnt 0
  yearsum 3 costsum 5 3 80
  newcostsum 5 newcnt END printf
  Avg. car age 3.1f yrs\n, \ 90 -
  (yearsum/NR) printf Avg. car cost 7.2f\n,
  \ costsum/NR printf Avg. newer car cost
  7.2f\n, \ newcostsum/newcnt

continued
41

• awk -f summary carsAvg. car age 13.2 yrsAvg.
  car cost 6216.67Avg. newer car cost 8750.00

42
3.10. Associative Arrays

• Arrays that use strings as indexes
• arraystring value
• Special for-loop for awk arrays
• for (elem in array) action

continued
43
manuf

• cat manuf manuf1END for
  (name in manuf) \ print name, manufname
  

continued
44

• awk -f manuf carshonda 1 fiat 1 volvo 1
  ford 4 plym 1chevy 3toyota 1

45
Sorted Output

• Sort by first column (i.e. by name)
• awk -f manuf cars sort
• Sort by second column (i.e. by number)
• awk -f manuf cars sort 1

46
4. sed

• Format
• sed list of ed commands file
• Read lines one at a time from the input file
• apply ed commands in order to each line
• write edited line to stdout
• ed is an old UNIX editor
• vi without full-screen mode
• did you think vi was tough )

47
4.1. Search and Replace

• The s command searches for a pattern (a regular
  expression), and replaces it with the new
  string
• s/pattern/new-string/g
• g means global (everywhere on line)

48
Examples

• sed s/UNIX/UNIX(TM)/g file new-file
• sed s// / file new-file
• put a tab at the start of every line (no g
  needed)
• sed s/ /\/g file new-file
• replace every sequence of blanks or tabs with a
  newline
• this splits the input into 1 word/line

continued
49

• whoad tty1 Sep 29 0714ron tty3 Sep 29
  1031td tty4 Sep 29 0836 who sed s/ .
  / /ad 0714ron 1031td 0836

replace a blank and everything that follows it
(as much as possible, including more blanks) up
to the last blank
50
More Information

• sed can use most ed commands, not just s
• See the entry on sed in Sobell, p.680-691

51
5. find

• Format
• find starting-directory matching-conditions-a
  nd-actions
• find searches all the directories below the
  starting directory.
• it carries out the specified actions on the files
  that match the specified conditions

52
Basic Example

• Assume we are in my home directory, and want to
  find the cars file (used in the awk examples)
• find . -name cars -print./teach/adv-unix/fi
  lters/cars

starting point
-name condition
-print action
53
5.1. Some Matching Conditions

• -name nm the filename is nm
• -type ty ty is a file type f file, d
  directory, etc.
• -user usr the files owner is usr
• -group grp the files group owner is grp

continued
54

• -atime n file was last accessed exactly n days
  ago
• -mtime n file was last modified exactly n days
  ago
• -size n file is exactly n 512-byte blocks
  long
• Can use or - to mean more or less.

55
5.2. Example Conditions

• -mtime 7 last modified more than 7 days
  ago
• -size 100 larger than 50K
• Anding conditions
• -atime 60 -mtime 120
• files last accessed more than 2 months ago and
  last modified more than 4 months ago

continued
56

• Oring Conditions
• \( -mtime 7 -o -atime 30 \)
• files last modified more than 7 days ago or last
  accedded more than 30 days ago
• Not
• -name \.dat \! -name gold.dat
• all .dat files except gold.dat

57
5.3. Some Actions

• -print display pathname of matching file
• -exec cmd execute cmd on file
• -ok cmd prompt before executing cmd on file
• Commands must end with \ and use to mean the
  matching file, e.g.
• -ok rm \

58
5.4. Examples

• find . -name \.c -print
• Starting from the current directory, display the
  pathnames of all the files ending in .c
• find . \( -name core -o -name junk \)
  -print -ok rm \
• Print the pathnames of all the core and junk
  files in the current directory and below, and
  prompt to remove them.

continued
59

• find /usr -size 100 -mtime 30 -exec ls -l
  \
• Display a long list of all the files under /usr
  larger than about 500K that have not been
  modified in a month.

60
5.5. Problems with Permissions

• A find over the entire filesystem will print many
  error messages when access is denied to other
  users directories.
• These error messages (sent to stderr) can be
  redirected to /dev/null (a UNIX black hole).

61
Example

• Search for a file/directory called zip anywhere
  below the root directory
• find / -name zip -printfind
  /exports/tmp/code/4210341 Permission
  deniedfind /exports/tmp/code/4210389
  Permission deniedfind /exports/home/suthon/priva
  te Permission deniedfind /exports/home/cj/mail
  Permission denied

continued
62

• Redirect standard errors to the black hole using
  2
• find / -name zip -print 2 /dev/null/exports/h
  ome/s4110068/project/zip/exports/home/s4110316/pr
  oject/zip/exports/home/s4110316/zip/exports/home
  /s4110316/zip/zip