Introduction to Unix (CA263) File Processing (continued) By Tariq Ibn Aziz

Guide to UNIX Using Linux, Third Edition 2 Objectives Use the pipe operator to redirect the output of one command to another command Use the cut, paste, tr and grep command to search for a specified pattern in a file Use the uniq command to remove duplicate lines from a file Use the comm and diff commands to compare two files Use manipulation and transformation commands, which include sed, tr, and pr Design a new file-processing application by creating, testing, and running shell scripts

Guide to UNIX Using Linux, Third Edition 3 Advancing Your File-Processing Skills Selection commands focus on extracting specific information from files

Guide to UNIX Using Linux, Third Edition 4 Advancing Your File Processing Skills (continued)

Guide to UNIX Using Linux, Third Edition 5 Using the Selection Commands Using the Pipe Operator – The pipe operator (|) redirects the output of one command to the input of another –The pipe operator can connect several commands on the same command line

Guide to UNIX Using Linux, Third Edition 6 Cut Command To extract various column or fields of data file or the output of the command. –Cut –cchars file –cut –c5- data –It will extract characters 5 through the end of the line of data and write the results to standard output.

Guide to UNIX Using Linux, Third Edition 7 Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8 root taziz dawn amin

Guide to UNIX Using Linux, Third Edition 8 Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8 | sort amin dawn root taziz

Guide to UNIX Using Linux, Third Edition 9 Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c10-16 console tty02 tty08 tty10

Guide to UNIX Using Linux, Third Edition 10 Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8,18- root Feb 24 08:54 taziz Feb 24 12:55 dawn Feb 24 09:15 amin Feb 24 15:35

Guide to UNIX Using Linux, Third Edition 11 Cut Command The –d and –f Option The -d and –f option are used with cut when you have data that is delimited by a particular character. The format of the cut command is as follow: cut –ddchar –ffields file Where dchar is the character that delimits each field of data, and fields specifies the field to be extracted from file. Field numbers start at 1.

Guide to UNIX Using Linux, Third Edition 12 Cut Command The –d and –f Option cut –ddchar –ffields file $cat phonebook tariq: :11 Driscoll Dr kalim: :12 Driscoll Dr imran: :13 Driscoll Dr hasam: :14 Driscoll Dr $cut –d: –f1 phonebook tariq kalim imran Hasam $

Guide to UNIX Using Linux, Third Edition 13 Cut Command Field are separated by tabs. $cat phonebook Alice Chebba Barbara Swingle Jeff Goldberg Liz Stachiw $cut –c1-15 phonebook Alice Chebba59 Barbara Swingle Jeff Goldberg2 Liz Stachiw775 $

Guide to UNIX Using Linux, Third Edition 14 Cut Command Field are separated by tabs, you should use the –f option to cut $cat phonebook Alice Chebba Barbara Swingle Jeff Goldberg Liz Stachiw $cut –f1 phonebook Alice Chebba Barbara Swingle Jeff Goldberg Liz Stachiw $

Guide to UNIX Using Linux, Third Edition 15 Paste Command Its opposite of cut command, cut break lines apart, and paste command put lines together. $ cat names Tubs Emanuel Lucy Ralph $ cat numbers (307) (212) (212)MH (212)BE $ paste names numbers Tubs(307) Emanuel(212) Lucy(212)MH Ralph(212)BE0-7741

Guide to UNIX Using Linux, Third Edition 16 Paste Command The –d option Its opposite of cut command, cut break lines apart, and paste command put lines together. $ paste –d'+‘ names numbers Tubs+(307) Emanuel+(212) Lucy+(212)MH Ralph+(212)BE0-7741

Guide to UNIX Using Linux, Third Edition 17 Paste Command The –s option The –s option tells paste to paste together lines separated by tab from the same file, not from alternate files. $ paste –s names TubsEmanuelLucyRalphFred $ paste –d' ' –s - Tubs Emanuel Lucy Ralph Fred

Guide to UNIX Using Linux, Third Edition 18 tr command The tr filter is used to translate characters from standard output. The general form of the command is: tr from-char to-char $ cat intro The UNIX operating system was pioneered by Ken $ tr e x < intro Thx UNIX opxrating system was pionxxrxd by Kxn

Guide to UNIX Using Linux, Third Edition 19 tr command You can translate colon into tab character to produce more readable output, by simply tacking tr command to the end of the pipeline. $ cut –d: -f1,6 /etc/passwd root:/ cron:/ bin:/ uucp:/usr/spool/uucp asg:/ $ cut –d: -f1,6 /etc/passwd |tr : '' root/ cron/ bin/ uucp/usr/spool/uucp asg/

Guide to UNIX Using Linux, Third Edition 20 tr command Octal Values of some ASCII characters CharacterOctal value Bell 7 Backspace10 Tab11 NewLine12 Linefeed12 Formfeed14 Carriage Return15 Escape33 $ date |tr ' ' '\12' Sun Mar 10 19:13:46 EST 1985

Guide to UNIX Using Linux, Third Edition 21 tr command $ cat intro The UNIX operating system was pioneered by Ken $tr '[a-z]' '[A-Z]' < intro THE UNIX OPERATING SYSTEM WAS PIONEERED BY KEN

Guide to UNIX Using Linux, Third Edition 22 tr command The –s Option You can use the –s option to tr to “squeeze” out multiple occurances of characters will be replaced by a single character. $ cat lotsofspaces This is an example of a file that contains a lot of blank spaces. $ tr -s' ' ' ' < lotsofspaces This is an example of a file that contains a lot of blank spaces.

Guide to UNIX Using Linux, Third Edition 23 tr command The –d Option tr can also be used to delete single characters from a stream of input. $ tr –d ' ' < intro TheUNIXoperatingsystemwaspioneeredbyKen

Guide to UNIX Using Linux, Third Edition 24 Using the grep Command Used to search for a specific pattern in a file, such as a word or phrase grep’s options and wildcard support allow for powerful search operations You can increase grep’s usefulness by combining with other commands, such as head or tail command Syntax: grep [-options] [pattern] [filename] Useful options includes -iignore case -llists only file names -ccounts the number of line instead of showing them -rsearches through files under all subdirectories

Guide to UNIX Using Linux, Third Edition 25 grep Example $ grep shell ed.cmd files, and is independent of the shell. to the shell, just type in a q.

Guide to UNIX Using Linux, Third Edition 26 grep example $ cat phone_book Alice Chebba Barbara Swingle Jeff Goldberg Liz Stachiw Susan Goldberg Tony Iannino $ $ grep Susan phone_book Susan Goldberg

Guide to UNIX Using Linux, Third Edition 27 grep –v options Example Print all lines that don’t contain Unix $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ $ grep –v 'Unix' intro environment for efficient program development

Guide to UNIX Using Linux, Third Edition 28 grep –i option Example Print all lines by ignoring upper and lower case letters $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –i 'Unix' intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX

Guide to UNIX Using Linux, Third Edition 29 grep [tT] option Example grep search for either upper or lower case letters T $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep '[tT]he intro The unix operating system was pioneered by Ken

Guide to UNIX Using Linux, Third Edition 30 grep -i option Example grep with –l just gives you a list of file that contains specified pattern $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –l 'Main goal' * intro $

Guide to UNIX Using Linux, Third Edition 31 grep -n option Example grep with –n gives you output of specified pattern with relative line number $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –n 'environment' intro 3:environment for efficient program development $

Guide to UNIX Using Linux, Third Edition 32 Using the uniq Command Uniq command removes duplicate lines from a file Compares only consecutive lines, therefore uniq requires sorted input uniq has an option that allows you to generate output that contains a copy of each line that has a duplicate Syntax: uniq [-option] [file1 > file2] Useful options includes -uoutputs only the lines of the source file that are not duplicated -doutputs only the copy of each lines that has a duplicate, and does not show unique line -iignores case -cstarts each line by showing the number of each instance

Guide to UNIX Using Linux, Third Edition 33 Uniq Command Example $ cat parts muffler shocks alternator battery radiator coil spark plugs coil $ uniq parts >inventory $ more inventory muffler shocks alternator battery radiator coil spark plugs coil $ Why coil is listed twice?

Guide to UNIX Using Linux, Third Edition 34 Uniq –u Command Example $ cat parts muffler shocks alternator battery radiator coil spark plugs coil $ uniq –u parts >inventory $ more inventory shocks alternator coil $ Why coil is listed twice? Ans: Two occurrences of coil are not together

Guide to UNIX Using Linux, Third Edition 35 Using the comm Command Used to identify duplicate lines in sorted files Unlike uniq, it does not remove duplicates, and it works with two files rather than one It compares lines common to file1 and file2, and produces three column output –Column one contains lines found only in file1 –Column two contains lines found only in file2 –Column three contains lines found in both files

Guide to UNIX Using Linux, Third Edition 36 Using the diff Command Attempts to determine the minimal changes needed to convert file1 to file2 The output displays the line(s) that differ Codes in the output indicate that in order for the files to match, specific lines must be added or deleted

Guide to UNIX Using Linux, Third Edition 37 Using the pr Command to Format Your Output pr prints specified files on the standard output in paginated form By default, pr formats the specified files into single- column pages of 66 lines Each page has a five-line header containing the file name, its latest modification date, and current page, and a five-line trailer consisting of blank lines

Guide to UNIX Using Linux, Third Edition 38 Using a Shell Script to Implement the Application Shell scripts should contain: –The commands to execute –Comments to identify and explain the script so that users or programmers other than the author can understand how it works Use the pound (#) character to mark comments in a script file

Guide to UNIX Using Linux, Third Edition 39 Running a Shell Script You can run a shell script in virtually any shell that you have on your system The Bash shell accepts more variations in command structures that other shells Run the script by typing sh followed by the name of the script, or make the script executable and type./ prior to the script name

Guide to UNIX Using Linux, Third Edition 40 Putting it All Together to Produce the Report An effective way to develop applications is to combine many small scripts in a larger script file Have the last script added to the larger script print a report indicating script functions and results