1. AWK One tool to create them all AWK Marcel Nijenhof marceln@pion.xs4all.nl http://pion.xs4all.nl/lezingen/awk_eth-0_2010.pdf Eth-0 11 Augustus 2010

2. Agenda Introduction Awk syntax Examples Unix tools: cat, grep, head, wc, tail, sort | uniq -c Password checker factorial function

3. Marcel Nijenhof Nllgg Chairman 10 years board member LPI Nederland Board member Proctor Proxy Employee Unix administrator

4. AWK Alfred Aho Brian Kernighan Peter Weinberger awk dates from, I think, 1977. It's by far the biggest software project that I have been involved with. There were three of us in that, and that's completely unworkable. Somehow, it's much easier working with two rather than three. It's harder to split things. There's more divergence of opinion, sometimes that's good because it means that the more things there but sometimes it means that it's not as cohesive as it might be. On the other hand it was very very nice to work with Al Aho and Peter Weinberger so I had no problem with that. Brian Kernighan, A Quarter Century of UNIX, page 102, ISBN 0-201-54777-5

5. How to use awk? ps -ef | grep httpd | grep -v grep | awk '{print $2}' ps -ef | awk '/[h]ttpd/ {print $2}'

6. Why awk? It's standard available on all unix systems Allowed in shell scripting E.g. perl, python is more powerful  But this is programming not shell scripting! It's very good at filtering lines in a file  Log file reporter  Checking configuration

7. Syntax Runs the complete script for every input line Split input lines in variables ($1, $2, $3,...) PATTERN { ACTION} PATTERNS (requirement)  There can be more then one patern  Special patterns BEGIN, END ACTIONS  Run for successful paterns  Awk programming code

8. An example awk -F: '/home/ {print $1}' /etc/passwd Separator = “:” $1 = login, $2 = password, $3 = uid PATTERN = /home/ Select every line containing home ACTION = {print $1}' Print the login name

9. PATTERN / / $5 ~ / / – awk -F: '$3 ~ /^0$/ {print $0}' /etc/passwd / / || / / Numeric compare – echo -e "a\nb\nc\nd" | awk 'NR%2==1 {print $0}' – Print the odd numbered lines (e.g. a, c) BEGIN, END – awk 'BEGIN { print “Hello world”}'

10. Awk programming code Many functions String  gsub, length, match, tolower, printf Math  +, -, *, /  sin, cos, sqrt, exp, log, rand User defined  function () { } See man awk!

11. Variables (built-in) Default variables NR: Line number NF: Number of fields on the line FILENAME: Name of the current file IFS: Input file separator - Changes line splitting OFS: Output file separator...

12. Variables (user) User variables Automatic created No special first character (e.g. the “$” from perl) A = “test” B = 3 Arrays/Hashes Index with key or string  a[1] = “foo”  a[“bar”] = 3

13. Unix tools (trivial tools) cat awk '{print $0}' grep awk '/ / {print $0}' head awk 'NR<=10 {print $0}' wc -l awk 'END {print NR}'

14. Unix tools (wc) { chars += length($0)+1 words += NF } END { printf (“%4i %4i %4i %s\n, NF, words, chars, FILENAME) } No multi file support Possible with check for change FILENAME

15. Unix tools (wc multi files) { if (ONAME != FILENAME && ONAME != "") { printf ("%4i %4i %4i %s\n", NR-t_lines-1, words, chars, ONAME) t_chars += chars t_words += words t_lines = NR-1 chars = 0 words = 0 } ONAME = FILENAME chars += length($0)+1 words += NF } END { printf ("%4i %4i %4i %s\n", NR-t_lines, words, chars, FILENAME) if (t_chars > 0) { t_chars += chars t_words += words printf ("%4i %4i %4i %s\n", NR, t_words, t_chars, "TOTAL") } }

16. Unix tools (tail) { line[NR%10] = $0 } END { for (i=NR-9; i<=NR; i++) { print line[i%10] } } Differences with “tail” Not as efficient as tail (reads the complete file). No “tail -f”

17. Unix tools (sort | uniq -c) { a[$0]++ } END { for (v in a) { printf ("%7i %s\n", a[v], v) } }

18. Check passwd file (duplicated uid) awk -F: ' { c_uid[$3]++ } END { for (u in c_uid) { if (c_uid[u] > 1) { printf ("%i\t%i\n", c_uid[u], u) } } }' /etc/passwd Compare this with the awk version of “sort | uniq -c”

19. Passwd & Group consistency check awk -F: ' FILENAME ~ /passwd/ { login[$1]++ pgid[$4]++ } FILENAME ~ /group/ { split ($4, g, ",") for (u in g) { if (login[g[u]] == 0) { printf ("Error in group %s no user %s\n", $1, g[u]) } } delete pgid[$3] } END { for (gid in pgid) { printf ("Error no primary group %i\n", gid) } }' /etc/passwd /etc/group

20. factorial function (recursive) function fac(n) { if (n == 1) { return (1) } else { return (n*fac(n-1)) } } BEGIN { print fac(5) } Of course this is a demo for recursive functions BEGIN { s=1; for (i=2; i<=5; i++) { s=s*i }; print s}

21. Question

22. Presentation http://pion.xs4all.nl/lezingen awk_eth-0_2010.pdf awk_eth-0_2010.odp Copyright: CC Some rights reserved The following items are not covered by cc  Tux  Nllgg logo  Lpi logo  Proxy logo  The citation of Brian Kernigham Note: Clipart from http://www.openclipart.org