1. 6/13/2016Course material created by D. Woit 1 CPS 393 Introduction to Unix and C START OF WEEK 3 (UNIX) 6/13/2016Course material created by D. Woit 1

2. Comment on quotes Enclosing characters in single quotes preserves the literal value of each character in quotes. Enclosing characters in double quotes preserves the literal value of all characters within the quotes with the exception of ‘$’, ‘`’, ‘\’ and when history is enabled ‘!’. Example: –varx=1 –echo $varx 1 –echo “$varx” 1 –echo ‘$varx’ –$varx

3. 6/13/2016Course material created by D. Woit 3 Cut, Paste Commands cut extract specified input columns paste combine input columns both take input from stdin or files –Examples: –cut -c8 cutexample # outputs 8th column of each line of myfile –cut -c5-7,25- # outputs columns 5,6,7,25... of each line of stdin –cut -f2 table # outputs second field of each line of file table (default field separator for –f switch is tab) –cut -d' ' -f3 # outputs field 3, fields delimited by one space (here field separator is changed from tab to space) –paste file1 file2 # linei to stdout contains linei of f1 followed by a tab # followed by linei of f2 cat –A shows tabs, newlines etc. expand can convert tabs to spaces 6/13/2016Course material created by D. Woit 3

4. 6/13/2016Course material created by D. Woit 4 HMWK 1.Create a file called stdnt-file with last-names in column 1, first- names in column 2, id numbers in column 3, userids in column 4. What combination of commands (and possibly temp files) can you use to create a file, stdnt-ids, that contains only the id numbers and userids from stdnt-file? 2. How can you combine commands (and possibly temp files) to create another file, stdnt1, that contains only the first and last students in stdnt-file? 3. How can you combine commands to find out exactly how long userid dwoit (or whoever you want) has been idle (has not been typing anything)? You will find the "w" command useful. It tells you who is logged in, and what they're doing. For an explanation do: man w. Note that "dwoit" (or whoever) will likely have multiple entries (one for each "terminal" connection), so your output may have multiple lines. 6/13/2016Course material created by D. Woit 4

5. 6/13/2016Course material created by D. Woit 5 Transformers and Translators sed (stream editor) Example change string unix into UNIX –sed -e "s/unix/UNIX/" sedexample –-e says the string following is an edit command – the "s" in the string says substitute last string for first But only its first occurrence in line – sedexample is name of file to edit – output to stdout (but could redirect with > or >>) – input is taken from stdin or files – Regular expressions (like in grep) can be used. 6/13/2016Course material created by D. Woit 5

6. 6/13/2016Course material created by D. Woit 6 sed (stream editor) cont. sed -e "s/dog/cat/g“ sedexample changes all occurrences of "dog" to "cat" on each line (g=global) sed -e "s/xx*/1234/" fx –changes first occurrence of x or xx or xxx etc on a line to 1234 in file fx –Character before *, i.e. x* can occur 0 or more times. sed -e "1,5s/HI/hi there/" –changes "HI" to "hi there" on lines 1-5 of stdin sed -e "/UNIX/ s/shell/Shell/" f1.txt –changes first "shell" to "Shell" only in lines containing the string "UNIX" sed -e "/unix/ d" < myfile #deletes all lines containing string unix sed -e "1,5d" < myfile delete lines 1-5 6/13/2016Course material created by D. Woit 6

7. 6/13/2016Course material created by D. Woit 7 sed (stream editor) cont. Sed commands can be submitted from a file e.g: File sedfile has two lines: –s/shell/Shell/ –s/c language/ C Language/ –And can be used as sed –f sedfile infile if we want to include '/' in the change, we use another character to delimit, such as ? Instead of /. – sed -e "s?his/her?their?g“ – sed -e "sXhis/herXtheirXg“ Note: sed can be invoked in sequence e.g. change A to B and B to y /home/dwoit> sed -e 's/A/B/' -e 's/B/y/‘ –A –Y 6/13/2016Course material created by D. Woit 7

8. 6/13/2016Course material created by D. Woit 8 tr (translator ) translates input, character by character, and writes to stdout given translation table specified by from-string and to-string uses a 1:1 correspondence between from/to strings tr ":" ";" <inf –changes all colons into semi-colons (uses file inf as input) tr "x" "AX" <inf –changes all blanks to "A" and all "x" to "X“ tr "A-Z" "a-z" outfile –changes all uppercase to lowercase letters (so it would make stdout a lowercase version of stdin) tr "a-zA-Z" "A-Za-z" <infile –changes all upper to lower, and lower to upper can use backslash escapes, \n (newline), \t (tab) etc tr " " "\n“ –changes all blanks to newlines 6/13/2016Course material created by D. Woit 8

9. 6/13/2016Course material created by D. Woit 9 tr (translator ) cont. can use octal codes, if begin with a \ (see man ASCII or ascii command) tr " x" "\012y" <inf –changes all blanks into newlines (\012 is octal newline) and all "x" to "y" can use Character Classes (see man grep) –tr '[:lower:]' '[:upper:]' tr -d "\n" <inf –removes all newline chars from inf (-d=delete) and displays on stdout tr -d "a-z" <inf –removes all lower case letters from inf tr -s " " " " <inf –(-s = squeeze) option will take all repeated occurrences of char1 and replace by just char2. In example above char1=“ “ and char2=“ “ –i.e., all strings of 1 or more blanks are replaced by just one blank 6/13/2016Course material created by D. Woit 9

10. 6/13/2016Course material created by D. Woit 10 tr (translator ) cont. More examples with –s squeeze option: tr -s "Ae" " Y" <inf –All strings of one or more "A"s are replaced by one space –All strings of one or more "e"s are replaced by one "Y" tr -s '[:punct:][:blank:]' '\n' –squeezes all punctuation and horizontal whitespace into one newline (so it puts each word of input on its own line) More information for tr command can be found in man pages 6/13/2016Course material created by D. Woit 10

11. 6/13/2016Course material created by D. Woit 11 Pipes A pipe | connects stdout of one command to stdin of another e.g., ls | more # lists all files in current dir in alpha order, one screen at a time ls -t | head # lists the 10 most recently modified files in this dir. ls -t ------------------> head ---------> screen redirection is also possible : ls -t | head >outfile see if string "IEEEtran" is anywhere in the first 10 lines of huge file bigfile.tex: – head < bigfile.tex | grep "IEEEtran" – big.file ------> head -------------> grep ------> screen – stdin stdout stdin stdout 6/13/2016Course material created by D. Woit 11

12. 6/13/2016Course material created by D. Woit 12 Pipes cont. last | head –see last few times someone has logged in (last's output is LONG) pipes contribute to shell's usefulness and flexibility no need to create new commands to perform a task– we can *combine* existing commands existing commands to perform more complex tasks no need for "temp" files 6/13/2016Course material created by D. Woit 12

13. 6/13/2016Course material created by D. Woit 13 Tee command saves the data passing through the pipe writes stdin to stdout *and* to a given file Example we want to change head < bigfile.tex | grep " IEEEtran" To head < bigfile.tex |tee first10| grep "IEEEtran" –Does exactly same, but also saves first 10 lines of bigfile.tex in file first10 6/13/2016Course material created by D. Woit 13

14. 6/13/2016Course material created by D. Woit 14 Example shell programs - #1 #!/bin/bash #source profs #prints names of profs logged in who | grep dwoit | cut -c1-8 | uniq who | grep eharley | cut -c1-8 | uniq who | grep jmisic | cut -c1-8 | uniq exit 0 #or who | egrep "dwoit|eharley|jmisic" | cut -c1-8 | sort –u Command who displays all users logged in Command uniq prevents repeated output of a line 6/13/2016Course material created by D. Woit 14

15. 6/13/2016Course material created by D. Woit 15 Example shell programs - #2 #!/bin/bash #source prof1 #prints names of a prof logged in #name supplied as command line argument who | grep $1 | cut -c1-8 See what happens if you use prof1 without an argument. How do you fix it? (hint: use quotes) who | grep “$1” | cut -c1-8 Corrected pgm prof1a will interpret lack of argument as a null string and print all users 6/13/2016Course material created by D. Woit 15

16. 6/13/2016Course material created by D. Woit 16 HMWK Use various UNIX commands and pipes to do the following (do not use any temp files): 1. List, in long form, all those files in your home directory that have read, write and execute perms for you, and whose name contains the string "txt". 2. Do the above question again, but list in short form (just the file names.) 3. List any lines in the first 10 lines of a file, myf, that contain the word "dog". (you should create different versions of myf to test your commands.) 4. Find all lines of file, myf, that contain all the words "cat", "dog" and "mouse", in any order, and start with the letter "A". 5. List all the files in your home dir that contain the string "so" somewhere within them. Next, just list the *number* of files in your home dir that contain the string "so" (you should use, among other commands, the command "wc -l", word-count (lines option), which you should look up in the man pages.) 6/13/2016Course material created by D. Woit 16

17. 6/13/2016Course material created by D. Woit 17 HMWK cont. 6. Use only ls, grep, cut, pipes to answer this question. What sequence of pipes and commands could you issue from your home dir to list, in alphabetical order, the absolute path names of all the directories in your file system (if you NEED to, use the sort command.) 7. List the names of any subdirectories of your home directory that have rwx permissions for user and group, and no permissions for others--note: list just the directory names in short form (the names only.) 8. List just names of all files/directories that have no extension; 9. List just names of all files/directories that contain no vowels at all. 6/13/2016Course material created by D. Woit 17

18. 6/13/2016Course material created by D. Woit 18 HMWK cont. 10. Write a shell program called nw that takes either zero or one command line arguments. – nw with no arguments should print out the 10 newest (most recently modified) files/dirs in the current directory. – nw -n (nw with one argument, -n) prints out the n newest files/dirs in the current directory. – e.g., nw -3 prints out the 3 most recently modified files/dirs – If the argument is incorrect, then your program is allowed to do unpredictable things! 11. List names in reverse alphabetical order, of directories that have "r" permissions for "other". (the "reverse" option of the sort command may be useful.) 12. List in alphabetical order all files in the current directory that have been edited exactly 3 days ago. Do not list the same file more than once. (the uniq command will eliminate duplicate lines). 6/13/2016Course material created by D. Woit 18

19. 6/13/2016Course material created by D. Woit 19 Foreground/Background Processing, execute commands in background with & /home/dwoit> find / -name firefox -print >find.out 2>/dev/null & /home/dwoit> # free to do other work /home/dwoit> grep "^the" bigfile.tex >grep.out 2>/dev/null & How to list the active processes? /home/dwoit> ps -l # lists processes, ids, status –Status can be R running, S suspended (waiting for an event to complete), D uninterruptable sleep, T stopped by job control signal... PID... CMD... 2498... find... 1483... grep... 3398... ps 6/13/2016Course material created by D. Woit 19

20. 6/13/2016Course material created by D. Woit 20 Foreground/Background Processing Note: ps x gives shorter list or alternatively –ps –u user A shell associates a job with each pipeline /home/dwoit> jobs – [1] Running find... & – [2] Running grep... & /home/dwoit> fg %1 # now in foreground--no prompt until done –^z # suspends foreground process (ctrl-z) /home/dwoit> bg %1 # starts find again in background /home/dwoit> kill %1 or /home/dwoit> kill 2498 # find cmd is killed (^c if fg process) might need guaranteed kill: –kill -9 2498 6/13/2016Course material created by D. Woit 20

21. 6/13/2016Course material created by D. Woit 21 Command History Editing for bash !! re-execute last command issued !-n re-execute current command minus n !cmd re-execute last command that started with string "cmd" ^str1^str2^ re-execute last cmd but substitute str2 for str1 history (or fc -l) lists last commands (each is numbered) 6/13/2016Course material created by D. Woit 21

22. 6/13/2016Course material created by D. Woit 22 Examples /home/dwoit> ls lab1.jav lab2.jav lab1.c393 lab2.c393 /home/dwoit> !! lab1.jav lab2.jav lab1.c393 lab2.c393 /home/dwoit> find. -name "*.jav" lab1.jav lab2.jav /home/dwoit> ^jav^c393^ lab1.c393 lab2.c393 /home/dwoit> history – 8 ls – 9 ls – 10 find. -name "*.jav" -print – 11 find. -name "*.c393" -print – 12 history 6/13/2016 Course material created by D. Woit 22

23. 6/13/2016Course material created by D. Woit 23 Examples cont. /home/dwoit> !10 lab1.jav lab2.jav History in editing: set -o vi makes command line editing use vi set -o emacs makes command line editing use emacs if set -o vi, then –esc-k brings back last command for editing (using vi) repeated "k"s move back in history, "j" move forward if set -o emacs, then –up-arrow brings back last command for editing (using emacs) repeated "up-arrows"s move back in history, "down-arrows" move forward 6/13/2016Course material created by D. Woit 23

24. 6/13/2016Course material created by D. Woit 24 HMWK When the "find" command tries to look in a directory you don't have permission to read, it prints an error message on stderr (try this out to see what happens). If you redirect stderr to stdout, then "find" will print both on stdout and you can pipe all the output. Use find, pipes, grep and wc to print the *number* of directories that "find" cannot read in your filesystem. -- Do the same thing, but have the number written into a file called num.no.read (using redirection). -- Repeat the previous question (output to file) but for the entire filesystem (i.e., start your "find" search at root)--but run the command in the background, or you will have to wait a long time for it to finish. If you want to logout before it's done, kill it first. 6/13/2016Course material created by D. Woit 24