1. 240-491 Adv. UNIX: Filters/41 Advanced UNIX v Objectives –to discuss five useful filters: tr, grep, awk, sed, and find 240-491 Special Topics in Comp. Eng. 1 Semester 2, 2000-2001 4. Filters (Part II, Sobell)

2. 240-491 Adv. UNIX: Filters/42 1. tr v format: tr [options] string1 [string2]  tr reads its standard input and translates each character in string1 to the corresponding character in string2

3. 240-491 Adv. UNIX: Filters/43 Examples $ echo 12abc3def4 | tr ’abcdef’ ’xyzabc’ 12xyz3abc4 $ echo 12abc3de4 | tr ’[a-c][d-f]’ ’[x-z][a-c]’ 12xyz3abc4 $ cat foo.txt | tr ’[A-Z]’ ’[a-z]’

4. 240-491 Adv. UNIX: Filters/44 $ tr ’\015’ ’ ’ file2 –\015 is carriage return $ cat mail.txt | tr -s ’ ป ’ ’ ’ > new-mail.txt – ป represents tab; could write \011 –-s means remove duplicates of string2 in output $ echo Can you read this? | tr -d ’aeiou’ Cn y rd ths?

5. 240-491 Adv. UNIX: Filters/45 “rot13” Text $ echo Gur chapuyvar bs gur wbxr vf... | tr ’[N-Z][A-M][n-z][a-m]’ ’[A-M][N-Z][a-m][n-z]’ The punchline of the joke is... Popular in 1970-1980’s.

6. 240-491 Adv. UNIX: Filters/46 2. grep v Format: grep [options] pattern [file-list] v Search one or more files, line by line, for a pattern (a regular expression). Actions taken depend on options.

7. 240-491 Adv. UNIX: Filters/47 Variants of grep  grep Uses basic RE pattern  fgrep Fast grep. Pattern can only be an ordinary string.  egrep Extended grep. Pattern can use full REs.

8. 240-491 Adv. UNIX: Filters/48 grep options v -cprint a count of matching lines v -iignore case in pattern during search v -llist filenames with match v -nprecede each matching line by a line number v -vprint lines that do not match pattern

9. 240-491 Adv. UNIX: Filters/49 Examples File testaFile testbFile testc aaabbaaaaaAAAAA bbbccbbbbb BBBBB ff-ffcccccCCCCC cccddddddd DDDDD dddaa continued

10. 240-491 Adv. UNIX: Filters/410 v $ grep bb testa aaabb bbbcc v $ grep -v bb testa ff-ff cccdd dddaa v $ grep -n bb testa 1: aaabb 2: bbbcc continued

11. 240-491 Adv. UNIX: Filters/411 v $ grep bb * testa: aaabb testa: bbbcc testb: bbbbb v $ grep -i bb *$ grep -i BB * testa: aaabbtesta: aaabb testa: bbbcctesta: bbbcc testb: bbbbbtestb: bbbbb testc: BBBBBtestc: BBBBB

12. 240-491 Adv. UNIX: Filters/412 Fancier Patterns v $ grep ’fun..ion’ file v $ grep -n ’^#define’ file v $ grep ’^#de[a-z]*’ file v $ egrep ’while|if’ *.c v $ egrep ’[0-9]+’ *.c

13. 240-491 Adv. UNIX: Filters/413 3. awk v format: awk program file-list awk -f program-file file-list  awk is a pattern scanning and action processing language v The action language is very like C.

14. 240-491 Adv. UNIX: Filters/414 Overview 3.1. Patterns & Actions 3.2. awk Processing Cycle 3.3. How awk Sees a Line 3.4. Pattern Expressions 3.5. ‘,’ Range Operator continued

15. 240-491 Adv. UNIX: Filters/415 3.6. Many Built-in Functions 3.7. BEGIN and END 3.8. First awk Program File: pre_header 3.9. Action Language 3.10. Associative Arrays

16. 240-491 Adv. UNIX: Filters/416 3.1. Patterns & Actions  An awk program consists of: pattern {action} pattern {action} :

17. 240-491 Adv. UNIX: Filters/417 3.2. awk Processing Cycle 1. Read next input line. 2. Apply all awk patterns sequentially. 3. If a pattern matches, do its action. 4. Go to step (1).

18. 240-491 Adv. UNIX: Filters/418 Example v $ cat cars plymfury77732500 chevynova79603000 fordmustang654510000 volvogl781029850 fordltd831510500 chevynova80503500 fiat60065115450 hondaaccord81306000 fordthundbd841017000 toyotatercel82180750 chevyimpala65851550 fordbronco83259500 continued

19. 240-491 Adv. UNIX: Filters/419 v $ awk ’/chevy/ {print}’ cars chevynova79603000 chevynova80503500 chevyimpala65851550 v $ awk ’/chevy/’ cars chevynova79603000 chevynova80503500 chevyimpala65851550 v $ awk ’/^h/’ cars hondaaccord81306000

20. 240-491 Adv. UNIX: Filters/420 3.3. How awk Sees a Line  awk views each line as a record consisting of fields separated by spaces.  Each field is referred to by a variable called $ : –$1, $2, $3, etc. –$0 refers to the whole line (record)  The current line number is stored in NR continued

21. 240-491 Adv. UNIX: Filters/421 v $ awk ’{print $3, $1}’ cars 77 plym 79 chevy 65 ford : 83 ford v $ awk ’/chevy/ {print $3, $1}’ cars 79 chevy 80 chevy 65 chevy

22. 240-491 Adv. UNIX: Filters/422 3.4. Pattern Expressions v Format: variable OP pattern  OP forms: –matching:~!~ –ariithmetic: = > –boolean:&&||! continued

23. 240-491 Adv. UNIX: Filters/423 v $ awk ’$1 ~ /h/’ cars chevynova79603000 chevynova80503500 hondaaccord81306000 chevyimpala65851550 v $ awk ’$1 ~ /^h/’ cars hondaaccord81306000 continued

24. 240-491 Adv. UNIX: Filters/424 v $ awk ’$2 ~ /^[tm]/ {print $3, $2, “$” $5}’ cars 65 mustang $10000 84 thundbd $17000 82 tercel $750 v $ awk ’$3 ~ /5$/ {print $3, $1, “$” $5}’ cars 65 ford $10000 65 fiat $450 65 chevy $1550 continued

25. 240-491 Adv. UNIX: Filters/425 v $ awk ’$3 == 65’ cars fordmustang654510000 fiat60065115450 chevyimpala65851550 v $ awk ’$5 <= 3000’ cars plymfury77732500 chevynova79603000 fiat60065115450 toyotatercel82180750 chevyimpala65851550 continued

26. 240-491 Adv. UNIX: Filters/426 v $ awk ’$5 >= “2000” && $5 = “2000” && $5 < “9000”’ cars plymfury77732500 chevynova79603000 chevynova80503500 fiat60065115450 hondaaccord81306000 toyotatercel82180750 v $ awk ’$5 >= 2000 && $5 = 2000 && $5 < 9000’ cars plymfury77732500 chevynova79603000 chevynova80503500 hondaaccord81306000

27. 240-491 Adv. UNIX: Filters/427 3.5. ‘,’ Range Operator v Format: pattern1, pattern2 v Select a range of lines. –the first line of the range matches pattern1 –the last line of the range matches pattern2 v May return several groups of lines continued

28. 240-491 Adv. UNIX: Filters/428 v $ awk ’/volvo/, /fiat/’ cars volvogl781029850 fordltd831510500 chevynova80503500 fiat60065115450 v $ awk ’NR == 2, NR ==4’ cars chevynova79603000 fordmustang654510000 volvogl781029850 continued

29. 240-491 Adv. UNIX: Filters/429 v $ awk ’/chevy/, /ford/’ cars chevynova79603000 fordmustang654510000 chevynova80503500 fiat60065115450 hondaaccord81306000 fordthundbd841017000 chevyimpala65851550 fordbronco83259500 three groups

30. 240-491 Adv. UNIX: Filters/430 3.6. Many Built-in Functions  length(str) length of string str length length of current line  split(strings, array, delimitor) split string into parts based on the delimitor, and place in array –split(“a bcd ef g1”, arr, “ “) continued

31. 240-491 Adv. UNIX: Filters/431 v $ awk ’length > 23 {print NR}’ cars 3 9 10

32. 240-491 Adv. UNIX: Filters/432 3.7. BEGIN and END  BEGIN {action} executed before first line is processed  END {action} executed after last line is processed l $ awk ’END {print NR, “cars for sale.”}’ cars 12 cars for sale

33. 240-491 Adv. UNIX: Filters/433 3.8. First awk Program File v $ cat pr_header # # pr_header # BEGIN { print “Make Model Year Miles Price” print “---------------------------------” } {print} continued

34. 240-491 Adv. UNIX: Filters/434  $ awk -f pr_header cars Make Model Year Miles Price --------------------------------- plymfury 77 732500 chevynova 79 603000 : : chevyimpala 65 851550 fordbronco 83 259500

35. 240-491 Adv. UNIX: Filters/435 redirect_out v $ cat redirect_out /chevy/ {print > “chev.txt”} /ford/ {print > “ford.txt”} END {print “done.”} v $ awk -f redirect_out cars done. $ cat chev.txt chevynova79603000 chevynova80503500 chevyimpala65851550

36. 240-491 Adv. UNIX: Filters/436 3.9. Action Language v Very C like: –var = expr –if (cond) stat1 else stat2 –while (cond) stat –for (expr1; cond; expr2) stat –printf “format” expr1, expr2,... –{ stat1 ; stat2;... ; statN } v User-defined variables do not need to be declared continued

37. 240-491 Adv. UNIX: Filters/437 v Long statements, conditions, expressions may need to be typed over several lines. v Use ‘\’ to hide newline: if ($3 > 2000 && \ $3 2000 && \ $3 < 3000) print $3

38. 240-491 Adv. UNIX: Filters/438 price_range v $ cat price_range { if ($5 5000 && $5 = 10000) $5 = “expensive” printf “%-10s %-8s 19%2d %5d %-12s\n”, \ $1, $2, $3, $4, $5 } continued

39. 240-491 Adv. UNIX: Filters/439 v $ awk -f price_range cars plymfury197773inexpensive chevynova197960inexpensive : : fordbronco198325please ask

40. 240-491 Adv. UNIX: Filters/440 summary v $ cat summary BEGIN { yearsum = 0 ; costsum = 0 newcostsum = 0 ; newcnt = 0 } { yearsum += $3 ; costsum += $5 } $3 > 80 { newcostsum += $5 ; newcnt++ } END { printf “Avg. car age: %3.1f yrs\n”, \ 90 - (yearsum/NR) printf “Avg. car cost: $%7.2f\n”, \ costsum/NR printf “Avg. newer car cost: $7.2f\n”, \ newcostsum/newcnt } continued

41. 240-491 Adv. UNIX: Filters/441 v $ awk -f summary cars Avg. car age: 13.2 yrs Avg. car cost: $6216.67 Avg. newer car cost: $8750.00

42. 240-491 Adv. UNIX: Filters/442 3.10. Associative Arrays v Arrays that use strings as indexes: –array[string] = value  Special for-loop for awk arrays: –for (elem in array) action continued

43. 240-491 Adv. UNIX: Filters/443 manuf v $ cat manuf {manuf[$1]++} END { for (name in manuf) \ print name, manuf[name] } continued

44. 240-491 Adv. UNIX: Filters/444 v $ awk -f manuf cars honda 1 fiat 1 volvo 1 ford 4 plym 1 chevy 3 toyota 1

45. 240-491 Adv. UNIX: Filters/445 Sorted Output v Sort by first column (i.e. by name): $ awk -f manuf cars | sort v Sort by second column (i.e. by number): $ awk -f manuf cars | sort +1

46. 240-491 Adv. UNIX: Filters/446 4. sed v Format: sed ’list of ed commands’ file v Read lines one at a time from the input file –apply ed commands in order to each line –write edited line to stdout  ed is an old UNIX editor –vi without full-screen mode –did you think vi was tough :)

47. 240-491 Adv. UNIX: Filters/447 4.1. Search and Replace  The ‘ s ’ command searches for a pattern (a regular expression), and replaces it with the new string: ’s/pattern/new-string/g’ –‘ g ’ means global (everywhere on line)

48. 240-491 Adv. UNIX: Filters/448 Examples v $ sed ’s/UNIX/UNIX(TM)/g’ file > new-file v $ sed ’s/^//’ file > new-file –put a tab at the start of every line (no g needed)  $ sed ’s/[ ][ ]*/\/g’ file > new-file –replace every sequence of blanks or tabs with a newline –this splits the input into 1 word/line continued

49. 240-491 Adv. UNIX: Filters/449 v $who ad tty1 Sep 29 07:14 ron tty3 Sep 29 10:31 td tty4 Sep 29 08:36 $ who | sed ’s/.* / /’ ad 07:14 ron 10:31 td 08:36 $ replace a blank and everything that follows it (as much as possible, including more blanks) up to the last blank

50. 240-491 Adv. UNIX: Filters/450 More Information  sed can use most ed commands, not just s  See the entry on sed in Sobell, p.680-691

51. 240-491 Adv. UNIX: Filters/451 5. find v Format: find starting-directory matching-conditions-and-actions  find searches all the directories below the starting directory. –it carries out the specified actions on the files that match the specified conditions

52. 240-491 Adv. UNIX: Filters/452  Assume we are in my home directory, and want to find the cars file (used in the awk examples): $ find. -name cars -print./teach/adv-unix/filters/cars $ Basic Example starting point -name condition -print action

53. 240-491 Adv. UNIX: Filters/453  -name nm the filename is nm  -type tyty is a file type: f = file, d = directory, etc.  -user usr the file’s owner is usr  -group grp the file’s group owner is grp continued 5.1. Some Matching Conditions

54. 240-491 Adv. UNIX: Filters/454  -atime n file was last accessed exactly n days ago  -mtime n file was last modified exactly n days ago  -size n file is exactly n 512-byte blocks long v Can use + or - to mean more or less.

55. 240-491 Adv. UNIX: Filters/455 5.2. Example Conditions  -mtime +7 last modified more than 7 days ago  -size +100 larger than 50K v “And”ing conditions: -atime +60 -mtime +120 –files last accessed more than 2 months ago and last modified more than 4 months ago continued

56. 240-491 Adv. UNIX: Filters/456 v “Or”ing Conditions: \( -mtime +7 -o -atime +30 \) –files last modified more than 7 days ago or last accedded more than 30 days ago v “Not” -name \*.dat \! -name gold.dat –all “. dat ” files except gold.dat

57. 240-491 Adv. UNIX: Filters/457 5.3. Some Actions  -print display pathname of matching file  -exec cmd execute cmd on file  -ok cmd prompt before executing cmd on file  Commands must end with \; and use {} to mean the matching file, e.g.: -ok rm {} \;

58. 240-491 Adv. UNIX: Filters/458 5.4. Examples v $ find. -name \*.c -print –Starting from the current directory, display the pathnames of all the files ending in “.c ” v $ find. \( -name core -o -name junk \) -print -ok rm {} \; –Print the pathnames of all the core and junk files in the current directory and below, and prompt to remove them. continued

59. 240-491 Adv. UNIX: Filters/459  $ find /usr -size +100 -mtime +30 -exec ls -l {} \; –Display a long list of all the files under /usr larger than about 500K that have not been modified in a month.

60. 240-491 Adv. UNIX: Filters/460 5.5. Problems with Permissions  A find over the entire filesystem will print many error messages when access is denied to other user’s directories.  These error messages (sent to stderr ) can be redirected to /dev/null (a UNIX “black hole”).

61. 240-491 Adv. UNIX: Filters/461 Example  Search for a file/directory called zip anywhere below the root directory: $ find / -name zip -print find: /exports/tmp/code/4210341: Permission denied find: /exports/tmp/code/4210389: Permission denied find: /exports/home/suthon/private: Permission denied find: /exports/home/cj/mail: Permission denied : : continued

62. 240-491 Adv. UNIX: Filters/462  Redirect standard errors to the black hole using 2> $ find / -name zip -print 2> /dev/null /exports/home/s4110068/project/zip /exports/home/s4110316/project/zip /exports/home/s4110316/zip /exports/home/s4110316/zip/zip $