Introduction to Unix – CS 21 Lecture 12

Lecture Overview A few more bash programming tricks The here document Trapping signals in bash cut and tr sed awk

The Here Document Redirects stdin to a specific set of text located inside the same file << COMMAND << MARKER Data MARKER

Example of Here

Why Is This Useful? Allows you to keep all relevant information in one file Example: Database you want to search Don’t need to clutter up your directory with unnecessary temporary files

Trapping Signals Catching a signal is also called trapping a signal You can tell bash programs what to do when they receive different signals Analogy: When a postcard arrives, what do I do?

The ‘trap’ Command Usage: trap ‘COMMAND’ Signals Example: trap ‘cat errorMsg’ 4 6 In order to prevent you from running a program forever, signal number 9 cannot be trapped

Example Of trap

Two Helper Filters cut Break individual lines apart tr Change characters into different characters

The ‘cut’ Command More precise control over individual lines Will cut out certain words from each individual line so they can be processed Usage: cut [FLAGS] FILE

Flags -d Delimiter -f Field number Example cut –d: -f3 myFile

Example Of Cut

The ‘tr’ Command Translate Change on a one to one basis characters from one thing to another Usage: tr ‘Set1’ ‘Set2’ Example: tr ‘abc’ ‘ABC’ < myFile

Example Of tr

Two More Powerful Tools sed Stream Editor awk Alfred Aho, Peter Weinberger, and Brian Kernighan

The ‘sed’ Command/Language Filter Like grep, sort, or uniq, it takes input and performs some operation on it to filter the output Usage: sed ‘Address Command’ Address specifies where to make changes Command specifies what change to make Example: sed ‘4d’ textFile

Address Specification Addresses could be line numbers or regular expressions No address – each line One address – only that line Two comma separated addresses – All lines in between ! – All other lines

Commands Available To sed a\ Append text c\ Replace text i\ Insert text before d Delete lines s Make substitutions

Examples

More Examples

Substitution Example Same syntax as vi

When Would You Want To Use sed? sed works on streams, so it is perfect to be placed in the middle of a pipe in order to change the output from one format to another Example: If a program always prints out 4 lines of junk for every good line, sed can be used to weed out the junk

Example

awk Answers the question: What do I do if I want to search for a pattern and actually use it? Combination of grep and commands Searches for some pattern or condition and performs some command on it Complete programming language Looks a lot like C syntactically Variables are declared bash style

Pattern And Command awk in its most basic form simply executes a command on all lines that match (or adhere to) a certain pattern Usage: awk ‘Pattern { Command }’ FILE Just like sed, if there is no pattern, then every line will be matched

Example

Different Ways To Run Awk awk ‘Pattern { Command }’ awk –f awkFile inputFile Since awk itself can be a complex language, you can store all the commands in a file and run it with the –f flag

Important awk Concepts Record Every line of an input file is a record The current record can be referenced with $0 Field Every word in a record is called a field Each field is numbered and can be referred to $1 is the first record, $2 is the second, etc.

Special Predefined awk Variables RS The character that acts as a record separator Default is the end of a line FS The character that acts as a field separator Default is whitespace (space, tab, etc) Can be redefined with the –F flag

Example

Other awk Variables NF = number of fields in the current record NR = Total number of records seen so far OFS = Output field separator ORS = Output record separator

BEGIN And END Blocks Two special patterns that can be matched BEGIN Commands are executed before any records are looked at END Commands are executed after all records are processed

Example

Awk Patterns /regular expression/ -> same as egrep Relational expression >, =, <=, == Pattern && pattern Pattern || pattern Pattern1 ? Pattern2 : pattern3 If Pattern1 is True, then Pattern2, else pattern 3 (pattern) ! Pattern Pattern1, pattern2

Example Patterns

Awk Actions Enclosed in { } () Grouping $ Field reference Increment, decrement ^ Exponentiation + - ! Plus, minus, not * / % Multiplication, division, and modulus

Control Flow Statements Inside of commands, you can have control flow if while for

If Syntax if (condition) { Statements } else { Statements }

While Syntax while (Condition) { Statements }

For Syntax for (Declaration ; Condition ; Increment ) { Statements } for ( j=0; j < 5; j++) { print “hello world” }

When Would You Want To Use awk? Whenever you want to search for some pattern and perform some action Example: I want to go through and calculate the average score on the Midterm

Example

Another Example Adding 12 points to everyone’s midterm score

Putting Them Together – awk and sed

awk Versus bash $ arguments Always enclose everything to awk in single quotes so they don’t get interpreted $1 to awk means something completely different than $1 to bash $1 in awk means first field $1 in bash means first command line argument

Next Time Looking at some of the string and mathematical awk functionality find Putting everything together The complete bash programming necessities Quiz 2 – Next Tuesday