Pipes, Filters, and Redirection

1
Chapter 7

• Pipes, Filters, and Redirection

2
Overview

• Will use standard input, standard output, and
  terminal as files
• Will learn to use cut and paste commands to
  manipulate information in files

3
Overview

• Will learn to use pipes, filters, and
  redirection to create pipelines, modify
  information, and determine where information is
  read from and written to

4
Overview

• Will learn how to use regular expressions and
  extended regular expressions to search for
  information in files using various ambiguous file
  references
• Will format and print a document using the pr
  and lpr commands

5
Overview

• Will learn why and how head and tail commands
  are used
• Will learn how to compare files
• Will learn how to compress, extract, read, and
  archive the contents of a file

6
Standard Input and Output The Terminal as a File

• The Shell directs where standard
• input is to come from and where
• standard output and standard
• error should go

7
Standard Input and OutputThe Terminal as a File

• Three standard files are involved when a command
  is executed in Bash
• Note Term file is misleadingnot true files but
  file descriptors

8
Standard Input and Output The Terminal as a
FileTable 7.1 File Descriptors p. 345
9
Standard Input and OutputThe Terminal as a
FileFigure 7.1 Standard Input and Output and the
Terminal p. 345
10
Standard Input and OutputThe Terminal as a File

• Several commands to identify terminal
• whowho options arguments
• who am Iwho am i
• ttytty

11
Standard Input and Output The Terminal as a
FileTable 7.2 who Options p. 346
12
Activity Using Standard Input, Output, and
Terminal as a File

• KEY CONCEPTS
• who commanddetermines who is logged in and file
  name for terminal
• ltCtrlgt D sends EOF signal
• Names of devices kept in /dev directory
• Command with argumentcopies contents to
  standard output
• Command with no argumentcopies standard input
  to standard output

13
Activity Using Standard Input, Output, and
Terminal as a File

• KEY CONCEPTS
• Pseudo device can be accessed as device but has
  no physical existence
• Pseudo terminal is used as a terminal emulator
  for X Window System and to telnet into a system
• In X Window System each terminal window opened
  is device for that window
• Pseudo device
• Not attached to a real physical port

14
Standard Input and OutputStandard I/O

• Standard I/O is the capability of the shell to
  control and direct program input, output, and
  error information

15
Standard Input and OutputStandard I/O

• Keying in a command in the shell user is provided
  with three file descriptors
• Standard Input
• Standard Output
• Standard Error

16
Standard Input and OutputStandard I/O

• Many commands do not yield output to the screen
  unless there is an error

17
Standard Input and OutputStandard I/O

• Redirection tells shell to read/write
  information from/to nonstandard devices
• Only works when command expects to send/receive
  information from standard devices

18
Standard Input and OutputStandard I/OFigure 7.2
Redirecting Standard Output p. 350
19
Standard Input and OutputStandard I/O

• Redirection Symbols
• gt redirects output of file
• lt redirects input
• gtgt redirects output of command without
  overwriting existing file

20
Activity Using Redirection Symbols

• KEY CONCEPTS
• Maximum number of files on a FAT floppy disk
  root directory 224
• Long file names or file names with spaces take
  up more than one entry in Root Directory Table
• Redirection can be used with ls as command
  writes results to screen (standard output)
• gt signredirects standard output somewhere else

21
Activity Using Redirection Symbols

• KEY CONCEPTS
• Filter descriptors can
• filter out message errors by sending them to
  null device
• redirect error message to a file
• gtgt used to append text to end of file
• vi took commands from standard input and altered
  file

22
Linux Files File Structures

• Types of Files on Linux System
• Ordinary (Regular)Files
• Contain information that is created or can be
  manipulated

23
Linux Files File Structures

• Types of Files on Linux System
• Special Files
• Block Files
• Processes data one block at a time
• Used for reading/writing data to devices
• Character Files
• Produces dataone character at a time
• Related to serial input/output devices

24
Linux Files File Structures

• Types of Files on Linux System
• Linked files
• Special directory entry that specifies one file
  points to another file (link)
• Whatever operation performed on linkactual file
  is what is operated on
• Directories
• Container for files

25
Linux Files File Structures

• Files can be structured in several ways
• ASCII files that have no structure
• Regarded as sequence of bytes
• Text filesFlexible but no way to retrieve
  specific information
• Record
• Data stored as sequence of fixed length and have
  internal structure
• Data can be retrieved and manipulated
• TreeRecords that do not have a fixed length

26
Linux Files File Structures

• Can manage/manipulate files with pipes, filters,
  and redirection

27
The cut and paste Commands

• cat command is used to append one file to another

28
The cut and paste Commands

• paste command
• Combines files
• Most useful when combining files with columns of
  information
• Syntax
• paste file1 file2

29
The cut and paste CommandsTable 7.3 paste
Options p. 357
30
The cut and paste Commands

• cut command
• Retrieves some, but not all fields in a file
• Can be used to remove specific columns or fields
  from a file
• Syntax
• cut options files...

31
The cut and paste CommandsTable 7.4 cut Options
p. 358
32
The cut and paste Commands

• When using options with cut command
• list is a sequence of integers
• Use a command to separate values
• Use a hyphen to specify a range

33
The cut and paste Commands

• Examples
• cut -f list -d char file 1 file 2
• cut -c list file1 file 2
• compare -f 1,3 with -f 1-3
• compare -c 1, 14 with -c 4-6

34
Activity Using the cut and paste Commands

• KEY CONCEPTS
• cat command used to combine files
• Data combined sequentially
• paste commandpastes information side by side
• Can extract fields from combined files

35
Activity Using the cut and paste Commands

• KEY CONCEPTS
• ASCII file
• cut command can cut data from fixed-length
  records or ASCII file
• Data appears to be in columnsactually straight
  line
• No fieldsdata separated by spaces not by
  assigned files
• Extract data by character position

36
Filters

• Filter commands manipulate information on input
  and produce modified output

37
Filters

• sort
• Sorts lines of text and writes output to the
  screen
• Default is ascending order
• Syntax
• sort options files

38
Filters Table 7.5 sort Options p. 363
39
Filters

• wc
• Counts number of bytes, words, and lines of
  input
• Multiple Files Listedprint a total line
• No Files Listedread standard input
• Syntax
• we options ...files...

40
FiltersTable 7.6 wc Options p. 364
41
ActivityUsing sort and wc

• KEY CONCEPTS
• Pressing ltCtrlgt Dno more data
• Sort command filters data and writes keyboard
  input to standard output device
• Older versionsort sequence uppercase words then
  lowercase words
• Later versionidentical words are placed
  together
• Both versions use -f option (intermixes
  lowercase and uppercase letters)

42
ActivityUsing sort and wc

• KEY CONCEPTS
• Sorting numbers
• Not treated mathematically
• Numbers are character not numeric value
• Character data sorted from left to right
• example Smith comes before Smythe
• Numeric data sorted by units
• example 6, 9, 22, 23, 31, 124, 222

43
Activity Using sort and wc

• KEY CONCEPTS
• However, Shell relies on ASCII sort sequence
• Sort order punctuation marks (including spaces),
  numbers letters (lowercase preceding uppercase)
• ASCII sort order from left to right
• Example 123 13 aa AA bb BB
• Example Carolyn Smith before Robert Nester
• Use sort -g to sort numbers in correct numerical
  order
• sort - r to sort in reverse or descending order

44
Activity Using sort and wc

• KEY CONCEPTS
• wc -l prints new line count
• blank line counts as line
• -w optioncounts words
• NOTE In all examples you have been managing
  files, not managing data

45
Pipes

• Pipes
• Allows standard output of one program to be used
  as standard input to next program
• Use Linux utility programs as building blocks
  for larger programs
• Use pipes and filters to redefine data

46
Pipes

• Pipes
• Symbol
• Syntax is command_1 options arguments
  command_2 options arguments command_3
• May redirect final output to a file or a device

47
Activity Using Pipes

• KEY CONCEPTS
• lsa command and not a file but because it
  writes standard output can pipe it in as input to
  a command
• --More-- at bottom of screenmore data
• ltEntergtscroll a line at a time
• ltSpaceBargtscroll a screen at a time
• qstop display and return to prompt
• lpr utility and also command that accepts input
  from either a file or standard input
• Any command with standard output can be piped to
  lpr

48
Introducing grep, egrep, and fgrep

• grep
• Searches file(s) for file containing a string of
  characters specified on command line
• Output written to standard output
• Syntax grep options pattern file or grep
  options -e patterns -f file file
• Searches for regular expressions

49
Introducing grep, egrep, and fgrepTable 7.7
Common Regular Expressions pp. 372
50
Introducing grep, egrep, and fgrep

• grep
• Also includes extended regular expression
• Regular expressions and special characters
• adds items to search pattern
• ? have zero or one occurrence of a string
• acts as or operator

51
Introducing grep, egrep, and fgrepTable 7.8
Extended Regular Expressions p. 373
52
Introducing grep, egrep, and fgrep

• Looking at syntax usually grep, egrep, and fgrep
  are grouped together

53
Introducing grep, egrep, and fgrep

• grepSearches for lines matching a given regular
  expression
• fgrepSearches only for fixed, uninterrupted
  strings rather than regular expressions
• egrepSearches for regular/extended regular
  expressions
• Extended version of grep
• Fastest of three commands

54
Introducing grep, egrep, and fgrepTable 7.9
grep, egrep, and fgrep Options pp. 373-374
55
Introducing grep, egrep, and fgrepTable 7.9
grep, egrep, and fgrep Options p. 357
56
Activity Using grep/egrep/fgrep

• KEY CONCEPTS
• grep locates regular expressions
• grep -E behaves same way as egrep
• egrep allows use of extended regular expressions
• Vertical bar indicates search for either item or
  both
• fgrep does not allow use of wildcards or regular
  expressions
• grep family of command writes to standard output

57
Activity Using grep/egrep/fgrep

• KEY CONCEPTS
• Primary use of pipes and filters is manipulating
  standard output/input commands
• Using pipes, filters, and redirection, the rules
  of OS are not suspended
• Use pipehas command on either side of pipe
• Taking standard output of a command using it as
  standard input to next command
• Not every command has standard output
• Redirection is an instead of action
• Used grep command with -c, -ci, and -ciw options

58
Formatting a Document with pr

• pr utility program
• Converts text files into a paginated, columned
  version
• If no files specified, pr read standard input
• By default, pr formats files into single-column
  pages of 66 lines
• To print in formatted formpipe formatted
  document to lpr
• Syntax pr options arguments

59
Formatting a Document with prTable 7.10 pp.
378-379
60
Activity Using pr

• KEY CONCEPTS
• pr command takes input and displays it in a
  formatted form
• To print formatted output pipe it to printer
• If header information has spaces need to enclose
  it with quotation marks

61
Introducing tee, tr, and ispell

• tee command
• Takes single input and sends output to standard
  output and to a file
• Syntax tee options files
• Minimal options
• -a which appends output to a file rather than
  overwriting file

62
Introducing tee, tr, and ispell

• tr command
• Translates and/or deletes characters from
  standard input, writing to standard input
• Takes input only from standard input
• Syntax is tr options string1 string2

63
Introducing tee, tr, and ispell

• Features of tr command
• Does not change contents of original file
• Used to substitute/delete characters specified
  by options and patterns
• Not possible to replace a single character with
  several characters
• Most common options used are -d and -s
• Often used to convert characters from lowercase
  to upper case or to remove incorrect character in
  a file

64
Introducing tee, tr, and ispellTable 7.11 tr
Options p. 381
65
Introducing tee, tr, and ispellTable 7.12
Special Characters Options p. 381
66
Introducing tee, tr, and ispell

• ispell command
• Interactive spell check program
• Can use program as a stand-alone or with an
  editor
• Syntax is ispell options files

67
Activity Using tree, tr, ispell

• KEY CONCEPTS
• The tr utility is used for translating every
  character
• If output is not saved to a file but only
  written to the screen then the original file is
  still intact
• To strip blank lines from a file use ASCII value
• The -s command is useful for removing a
  character that is repeated several times

68
Introducing head and tail

• Head and tail command display specific number of
  lines at top (head) or bottom (tail) of a file

69
Introducing head and tail

• Syntax
• head options files
• tail options files

70
Introducing head and tailTable 7.13 head Options
p. 387
71
Introducing head and tailTable 7.13 tail Options
p. 387
72
Activity Using head and tail

• KEY CONCEPTS
• Used head and tail command (with and without
  options) to display specific lines in a file

73
Introducing uniq, diff, comm

• uniq command
• File must be sorted for command to work
• Displays a file while skipping any adjacent
  lines
• Power is ability to remove duplicate lines from
  a sorted file
• Syntax uniq options file1 file 2

74
Introducing uniq, diff, and commTable 7.15 uniq
Options p. 389
75
Introducing uniq, diff, and comm

• diff command
• Compares two filesdisplays differences
• Does not change either file
• Displays actions needed to convert first file
  into second
• Compares contents of directories
• Syntax
• diff options diroptions file1 file 2

76
Introducing uniq, diff, and comm

• comm command
• Compares lines common to sorted files (file 1
  file2)
• Output displayed in three columns
• Syntax comm options file1 file 2
• Useful option is column number (-1, -2, or -3)
• If one of numbers used only that column will
  print

77
Introducing uniq, diff, and comm

• comm command selects duplicate or unique lines
  between two sorted files
• uniq command selects duplicate or unique lines
  within same sorted file

78
Activity Using uniq, diff, and comm

• KEY CONCEPTS
• Used uniq command to display, on standard output
  only, nonduplicated lines
• Use -u option to see lines in file that are not
  duplicated
• Use -d option to see one copy of each duplicated
  line
• Use gt symbol to save output to a file

79
Activity Using uniq, diff, and comm

• KEY CONCEPTS
• Used diff command to show where to make changes
  in a file to make one file identical to another
• lt symbol indicates that this line is from second
  file
• Can be used with directories

80
Activity Using uniq, diff, and comm

• KEY CONCEPTS
• Used comm command to show differences between
  files
• Use option -3 to see only differences between
  files
• Use -2 option to see what lines the two files
  have in common

81
File Compression and Archiving

• Compression
• Allows files to take up less space on disk
• Saves disk space
• Less sending/downloading

82
File Compression and Archiving

• Compressed (zipped) file
• One or more files and/or directories stored in a
  single file
• Uses less disk space than originals
• Utilities used to compress files
• gzip
• bzip2
• zip

83
File Compression and Archiving

• Archived file
• Group of files and directories stored in one
  file
• Takes up same amount of space as original files
• Files are easier to move or back up
• Utility program used for archiving tar

84
File Compression and Archiving

• Compressed (zipped) files cannot be archived
• Archived files can be compressed

85
File Compression and Archiving

• File Roller
• Red Hart Linux graphical utility
• Can be used to compress, decompress, and archive
  files

86
File Compression and Archiving Tools

• bzip2
• Compresses file/files
• Compressed file renamed to filename.bz2
• Syntax bzip2 options filenames . . .
• bunzip2
• Used to uncompress file/files
• Syntax gunzip2 options filenames . . .

87
File Compression and Archiving Tools

• bzip2 family of commands
• bzip2
• bunzip2
• bzcat
• bzip2recover

88
File Compression and Archiving Tools Table 7.16
bzip Options p. 395
89
File Compression and Archiving Tools

• gzip
• Compresses a file or files
• Compressed file renamed filename gz
• Syntax gzip options files
• gunzip
• Uncompresses a file or files
• Takes all of gzips options
• Syntax gunzip options files

90
File Compression and Archiving Tools

• zcat
• Reads compressed files compressed with gaip and
  writes them to standard output
• Identical to gunzip -c and uses options -fhLv
• Syntax zcat options files

91
File Compression and Archiving Tools Table 7.17
gzip, gunzip, and zcat Options p. 396
92
File Compression and Archiving Tools Table
7.17gzip, gunzip, and zcat Options p. 396
93
File Compression and Archiving Tools

• zip and unzip utilities are compatible with pkzip
  and pkunzip programs in Windows world
• Original file remains in place
• Creates new file that contains contents of files
  that are compressed

94
File Compression and Archiving ToolsTable 7.18
File Compression Tools p. 397
95
File Compression and Archiving Tools

• tar
• Used for archiving file/files
• Often compressed with gzip
• Used to create single file that when unpacked
  creates a directory structure
• Software that has been processed by tar and
  compressed by zgip usually has extension .tar,
  gz, or tgz
• Syntax
• tar options tarfile other-files

96
File Compression and Archiving Tools Table 7.19
tar Options p.397
97
Activity Using bzip2, bunzip2, gzip, gunzip,
zcat, tar, and File Roller

• KEY CONCEPTS
• Compressed files
• have .bz2 extension
• no longer exist in home directory with original
  name
• To see size of file when uncompressed pipe
  output from zcat to wc command
• Use -1 option for full details
• Keep original file and still have compressed
  filepipe output of cat command to gzip command
• File seen in redcompressed/archived file

98
Activity Using gzip, gunzip, zcat, and tar

• KEY CONCEPTS
• Can see contents of .gz file (must be text file)
  with zcat command
• Use tar command to create an archive of multiple
  files in a single file
• can be used to uncompress file
• c optioncreates tar file
• v optionsee what work is being done
• f optionarchive file to tar file
• ls commandto see size of file
• x optionextracts archived file