1. 2000 Copyrights Danielle S. Lahmani UNIX Tools G22.2245-001, Fall 2000 Danielle S. Lahmani email: lahmani@cs.nyu.edu Lecture 5

2. 2000 Copyrights Danielle S. Lahmani Overview: Unix Filters finish discussing /bin/csh features sort grep/ fgrep/ egrep find cmp/diff uniq, tr tar/cpio

3. 2000 Copyrights Danielle S. Lahmani /bin/csh: history mechanism For numbered commands –set prompt = '\! %’ storage of commands –set history = 40size of history file –set savehist=32save 32 commands across sessions

4. 2000 Copyrights Danielle S. Lahmani /bin/csh: hist mech (continued) command reexecution –!!reexecute previous command –!number –!prefix –!?substring command editing –% !even :s/pat1/pat2 substitute pat1 with pat2 –%^pat1^pat2reexecution of previous command substituting pat1 with pat2

5. 2000 Copyrights Danielle S. Lahmani ACCESS PORTION OF A FILENAME Modifierpart of fileportion of filename returned :hheadfilename minus trailing path :rrootfilename minus trailing *.suffix :eextensiontrailing.* suffix :ttailfilename minus leading dir path

6. 2000 Copyrights Danielle S. Lahmani /bin/csh: CONTROL STRUCTURES foreach ……. end if then else endif switch case … endsw while … end onintr [-l label] allows to specify a label to jump to when shell receives a SIGINT

7. 2000 Copyrights Danielle S. Lahmani /bin/csh: PIPING Process1 |& process2 Redirects standard output and standard error from process 1 to process 2

8. 2000 Copyrights Danielle S. Lahmani /bin/csh:MULTIPLE REDIRECTION Cmd >& filesend both standard output and standard error to file Cmd >&! file same as above, even in noclobber is set Cmd >>& file append standard error and standard output to end of file Cmd1 | & cmd2 pipe standard error together with standard output (cmd > file1) >& file2 send standard output to file1; send standard error to file2

9. 2000 Copyrights Danielle S. Lahmani /bin/csh: JOB CONTROL Same as ksh, plus stopsuspend background process suspendsuspend foreground process notifynotify when current job changes status

10. 2000 Copyrights Danielle S. Lahmani /bin/csh: DIRECTORY STACK pushd [+number|name] pushd pushes the directory onto the directory stack popd [+number] popdpops a directory from the directory stack dirs: lists the current directory stack.

11. 2000 Copyrights Danielle S. Lahmani Effective and efficient shell prog reference: Unix shell Tutorial G.Snyder J.R. Mashey Bell Labs Minimize number of processes generated Find out which cmds are built-in and which are not.  Number of data bytes accessed  Minimize Directory searches  Optimize Directory search order and the PATH variable  Minimize arithmetic and character processing  Avoid command substitution when you can  Move loop invariant outside of loop, especially for command substitution

12. 2000 Copyrights Danielle S. Lahmani sort utility sort sorts a file in ascending or descending order based on one or more sort fields, and works as follows : sort [options]{filename}*

13. 2000 Copyrights Danielle S. Lahmani sort utility : common options sort filesort file by line, in ASCII order sort -r reverse normal order sort -n sort in numeric (arithmetic) order starting at the beginning of line sort -nr sort in reverse numeric order

14. 2000 Copyrights Danielle S. Lahmani sort utility: common options (continued) sort -f (“fold”) sort in alphabetical order ignoring case sort -o file save output to filename sort -u suppresses all but one of each group if lines that are identical in the sort fields

15. 2000 Copyrights Danielle S. Lahmani sort utility : common options (continued) sort [+m] [-n] sort starting at m+1 field and sort up to field n non- inclusive. If n is not defined, then it is assumed until the end of the line. The first field is numbered 0. sort -k field_start, field_end sort starting at field_start and ending at field_end inclusive. field numbers with this option start at 1.

16. 2000 Copyrights Danielle S. Lahmani grep/fgrep/egrep utilities grep [option] pattern {filename}* grep searches the named files or standard input and prints each line that contains an instance of the pattern. grep interprets the same regular expressions as ed(1)

17. 2000 Copyrights Danielle S. Lahmani grep/egrep/fgrep utilities g/re/p global regular expression print –regular expressions are specified by giving special meanings to certain characters.

18. 2000 Copyrights Danielle S. Lahmani grep/egrep/fgrep utilities ^ and $ "anchor" the pattern to the beginning (^) or end ($) of the line. –regular expressions metacharacters overlap with shell metacharacters, so it's always a good idea to enclose grep patterns in single quotes, to suppress interpretation by the shell. $ grep From $MAIL locates lines containing From in your mailbox $ grep '^From'$MAIL prints lines that begin with From in your mailbox

19. 2000 Copyrights Danielle S. Lahmani grep utility grep supports character classes much like those in the shell. –[a-z] matches any lower case letter –[^….] matches any character except those in the class –[^0-9] matches any non-digit –A period '.' is equivalent to the shell's ? it matches any single character.

20. 2000 Copyrights Danielle S. Lahmani grep examples print lines that begin with mary –$ who | grep '^mary' print lines that end with mary –$ who | grep 'mary$’ list files others can read and write –$ls -l | grep '^.......rw’

21. 2000 Copyrights Danielle S. Lahmani grep examples: the closure operator * applies to previous character or metacharacter ( including a character class) in expression grep 'abc*' matches ab followed by zero or more c's grep 'ab[a-z]*' matches ab followed by any number of lower case letters

22. 2000 Copyrights Danielle S. Lahmani grep utility grep '^[^:]*::' matches beginning of line, zero or more non- colons followed by a double colon. Inside the brackets, the ^ means not. no grep regular expression matches a newline; the expressions are applied to each line individually.

23. 2000 Copyrights Danielle S. Lahmani grep/fgrep/ egrep utilities Example: grep '^[^aeiou]*a[^aeiou]*e$' foo –Looks for zero or more non-vowels followed by a, followed by zero or more non vowels followed by e at the end of the line –Would match dsdakjkjkje … \ turns off meaning of special character that follows: –$ grep \' foofind all ' apostrophes in file foo

24. 2000 Copyrights Danielle S. Lahmani Common grep options -n prints line numbers $ grep -n variable *.[ch]locate variable in C source -v inverts the sense of the test $ grep -v From fooprint all lines that do not contain From in file foo -i makes lower case in pattern match either case in file $ grep -i mary $HOME/bin/phone-book

25. 2000 Copyrights Danielle S. Lahmani grep common options (continued) -c prints only a count of matched lines $grep -c /bin/csh /etc/passwd -l list filenames but not matched lines $ grep -l '^#include' /usr/include/*

26. 2000 Copyrights Danielle S. Lahmani egrep is an extended grep that works on extended regular expression,accepts: + one or more occurrences ? zero or one occurrence pat1| pat2 "or" op matches on either pat1 or pat2 ( r) regular expression r, can be nested (xy)* matches any of the empty string, xy, xyxy, xyxyxy and so on

27. 2000 Copyrights Danielle S. Lahmani fgrep/ egrep utilities fgrep searches for many literal strings simultaneously. It does not work with *. Both, fgrep and egrep have -f option, to read patterns stored in a file. In file, newlines separate patterns to be searched for simultaneously.

28. 2000 Copyrights Danielle S. Lahmani TABLE of Regular expressions for grep and egrep cany-non special character c matches itself \cturns off any special meaning of character c ^beginning of line $end of line.matches any single character […]any one of characters in …; [^…]any character not in …

29. 2000 Copyrights Danielle S. Lahmani TABLE of Regular expressions for grep and egrep (continued) r *zero or more occurrences of r r+one or more occurrences of r (egrep only) r?zero or one occurrences of r (egrep only) r1r2r1 followed by r2 r1| r2r1 or r2 ( egrep only) (r)nested regular expression r (egrep only)

30. 2000 Copyrights Danielle S. Lahmani find utility find pathlist expression find recursively descends through pathlist and applies expression to every file. Expression can be (subset): -name pattern true if file name matches pattern. Pattern mayinclude metacharacters such as *, must be in quotes to suppress shell interpretation.

31. 2000 Copyrights Danielle S. Lahmani find utility (continued) -perm mode find files with given access mode, mode must be in octal. -type ch file type ch (c=character, b=block, f for plain file, etc..) -user userid/username true if owner is userid or username -group groupid/groupname true if group of file is groupid or groupname

32. 2000 Copyrights Danielle S. Lahmani find utility (common options) !expression returns the logical negation of expression -print prints out the name of the current file Op1 a op2 matches both patterns op1 and op2 Op1 o op2 matches either op1 or op2

33. 2000 Copyrights Danielle S. Lahmani find utility (continued) -exec cmd executes cmd,where cmd must be terminated by an escaped semicolon (\;). If you specify {} as a command line argument, it is replaced by the name of the current file just found. Exec executes command once per file. Example: find -name "*.o" -exec rm {} \;

34. 2000 Copyrights Danielle S. Lahmani find utility (continued) find. -type f -print | xargs grep From -type f for files -print to print them out xargs works on every file found In general, xargs applies a utility (in this case grep) to one or more arguments with the remaining arguments coming from a single line of the standard input. xargs splits the standard input into a bunch of files that grep applies to.

35. 2000 Copyrights Danielle S. Lahmani cmp: comparing two files cmp: tests two files for equality. –finds the first byte that differs between two files –cmp -ls file1 file2 [offset1] [offset2 ] –-s causes all output to be inhibited –-l displays byte offset and values of every byte that doesn't match

36. 2000 Copyrights Danielle S. Lahmani diff: comparing two files diff: compares two files and outputs a description of their differences diff -i -Dflag file1 file2 - I makes diff ignore the case of the lines -D generates output designed for the C preprocessor

37. 2000 Copyrights Danielle S. Lahmani uniq: removing duplicate lines uniq [options] [inputfile [outputfile]] removes duplicate adjacent lines from inputfile sending one copy of each line to outputfile or standard output often used as a filter.

38. 2000 Copyrights Danielle S. Lahmani uniq: removing duplicate lines Options can be: - c print each line once, counting instances of each -d print duplicate lines once, but no unique lines - u print only unique lines ( no copy of duplicate entries is kept) -n ignore first n fields of a line. (fields are separated by space or tab)

39. 2000 Copyrights Danielle S. Lahmani tr:translating characters tr maps maps the characters in a file from one character set to another –example: –translate all lowercase characters to uppercase –tr a-z A-Z < inputfile –deletes all a-c characters –tr -d a-c < inputfile