1 Unix Seminar #1 T.J. Borrelli Lecturer for CS and NSSA February 6th, 2009

2 Processes.1 Types of processes: Types of processes: Foreground Foreground Shell waits for task to complete (before next prompt) Shell waits for task to complete (before next prompt) Task is connected to terminal Task is connected to terminal Shell is task’s parent Shell is task’s parent Background Background Shell does not wait for task to complete Shell does not wait for task to complete Task is connected to terminal Task is connected to terminal Shell is task’s parent Shell is task’s parent Daemon Daemon Shell does not wait for task to complete Shell does not wait for task to complete Task is not connected to terminal Task is not connected to terminal Shell is not task’s parent (parent = PID 1) Shell is not task’s parent (parent = PID 1)

3 Try and Learn Start the following in the background Start the following in the background sleep 30 & sleep 30 & When your prompt comes back, type ps When your prompt comes back, type ps

4 Job Control.1 Shows you the processes that are running in the background, or suspended in the fore/background Shows you the processes that are running in the background, or suspended in the fore/background Ideal for the ‘multitasking’ Ideal for the ‘multitasking’ Can have several tasks running at once Can have several tasks running at once Can switch among them at will Can switch among them at will Stop one task Stop one task Restart another Restart another Can view stopped tasks Can view stopped tasks Can kill tasks Can kill tasks

5 Job Control.2

6 Job Control.3 Useful commands: Useful commands: ^Z (ctrl-z) ^Z (ctrl-z) Stops a foreground job Stops a foreground job Job is frozen in place until controlled Job is frozen in place until controlled bg bg Places the most recently stopped job in the background Places the most recently stopped job in the background As though you had typed ‘&’ at end of the line As though you had typed ‘&’ at end of the line Can use bg %N to move job N to background if there is more than one stopped job (NOTE: the number after the % is not a process ID, it is a job number) Can use bg %N to move job N to background if there is more than one stopped job (NOTE: the number after the % is not a process ID, it is a job number) jobs jobs List existing fore/back-ground jobs List existing fore/back-ground jobs Each appears with a job number Each appears with a job number Job number IS NOT THE SAME AS process id Job number IS NOT THE SAME AS process id

7 Job Control.4 More job control commands More job control commands fg fg Bring the most recently stopped job OR background process to the foreground Bring the most recently stopped job OR background process to the foreground Can use fg %N to bring job N to the foreground if there is more than one stopped job (NOTE: the number after the % is not a process ID, it is a job number Can use fg %N to bring job N to the foreground if there is more than one stopped job (NOTE: the number after the % is not a process ID, it is a job number kill %2 kill %2 Kill job 2 … any legitimate job number may be used Kill job 2 … any legitimate job number may be used Can also … kill Can also … kill Kills by process ID Kills by process ID kill kills with extreme prejudice kill kills with extreme prejudice ^C (ctrl-c) ^C (ctrl-c) Control-C kills the foreground job Control-C kills the foreground job

8 Simple Shell Script.1 and Try and Learn Using ‘vi’ create a file named ‘firstscript’ Using ‘vi’ create a file named ‘firstscript’ Once created … save the file Once created … save the file At the prompt, type bash firstscript to run your script At the prompt, type bash firstscript to run your script #!/bin/bash # First shell script echo "List my home directory” ls -l

9 Simple Shell Script.2 What is a script? What is a script? Text file Text file First line must be: #!/bin/bash First line must be: #!/bin/bash Contains one Unix command per line Contains one Unix command per line Lines that begin with ‘#’ are comments (except that first one) Lines that begin with ‘#’ are comments (except that first one) When run, the commands are run in the order they appear in the script file When run, the commands are run in the order they appear in the script file

10 Script Execution.1 Run the script with the shell Run the script with the shell bash scriptname bash scriptname Run the script as an executable command Run the script as an executable command Type ‘ man chmod ’ to learn about chmod Type ‘ man chmod ’ to learn about chmod Try: chmod u+x scriptname./scriptname Try: chmod u+x scriptname./scriptname NOTE: no need to invoke the shell explicitly NOTE: no need to invoke the shell explicitly The shell to use is found in the #!/bin/bash line of the script The shell to use is found in the #!/bin/bash line of the script

11 Script Execution.2 Scripts are run in subshells Scripts are run in subshells Create a script named myps Create a script named myps Type: ps; bash myps Type: ps; bash myps What is the value of PID and what processes are running? What is the value of PID and what processes are running? #!/bin/bash echo " My PS " echo " PID=$$ " ps lx

12 Script Execution.3 When running a script, the shell starts a When running a script, the shell starts a shell Fork (copy) Input from terminal Input from script copy of itself called a subshell The subshell reads commands directly from the script file The subshell reads commands directly from the script file

13 Environment.1 Scripts execute in an “environment” Scripts execute in an “environment” Environment = state information Environment = state information Includes: Includes: Open files Open files Current directory Current directory User/group User/group Subshells inherit their environment from the shell that starts them Subshells inherit their environment from the shell that starts them

14 Environment.2 Create two scripts called firstScript and secondScript Create two scripts called firstScript and secondScript #!/bin/bash # This is called secondScript echo secondScript echo Working Directory pwd echo " " echo User/group id Make both of these executable and run firstScript Make both of these executable and run firstScript Note: your directory and “id” are the same inside secondScript as in firstScript Note: your directory and “id” are the same inside secondScript as in firstScript #!/bin/bash # This is called firstScript echo firstScript echo Working Directory pwd echo " " echo User/group id./secondScript

15 Environment.3 NOW: rerun firstScript as:./firstScript >output NOW: rerun firstScript as:./firstScript >output The “>output” sends the output to the file output The “>output” sends the output to the file output This is called redirection as it redirects the output to the file This is called redirection as it redirects the output to the file NOTE: the output of the secondScript is redirected to the file as well that of firstScript NOTE: the output of the secondScript is redirected to the file as well that of firstScript

16 Simple Shell Variables.1 Assignment Assignment VAR_NAME="a string" VAR_NAME="a string" NOTE: name is all upper case … not a requirement, a convention to make variables stand out in scripts NOTE: name is all upper case … not a requirement, a convention to make variables stand out in scripts NO spaces around the ‘=‘ … required NO spaces around the ‘=‘ … required The string is inside quotes “…” … required The string is inside quotes “…” … required NO semicolon at end of line … not allowed NO semicolon at end of line … not allowed Printing Printing echo "$VAR_NAME" echo "$VAR_NAME" NOTE: the name is preceded by ‘$’ … required NOTE: the name is preceded by ‘$’ … required The $VAR_NAME is in quotes … required sometimes The $VAR_NAME is in quotes … required sometimes

17 Simple Shell Variables.2 variables are not in the environment variables are not in the environment i.e., they are not inherited by subshells i.e., they are not inherited by subshells Variables can be placed in the environment by exporting them Variables can be placed in the environment by exporting them export VAR_NAME export VAR_NAME Exports without setting the value Exports without setting the value export ANOTHER_VAR="a string" export ANOTHER_VAR="a string" Exports and sets the value at once Exports and sets the value at once Modify firstScript and secondScript to try this out Modify firstScript and secondScript to try this out

18 The Command Line.1 The shell processes commands line-by-line The shell processes commands line-by-line Each line is read Each line is read Then parsed: separated into individual ‘words’ Then parsed: separated into individual ‘words’ Then executed: the first ‘word’ is the command to execute and the remaining words (if any) are arguments to the command Then executed: the first ‘word’ is the command to execute and the remaining words (if any) are arguments to the command The parsing step can be altered by the use of special characters called meta characters The parsing step can be altered by the use of special characters called meta characters

Special characters used to represent some other meanings &Processes a job in background sh MyJobInBackground & ;allows for multiple commands on line echo –n “Enter name: ” ; read Name > Redirects output ls –l > MyFiles >> Same except appends to end of file cd MyOtherDir ; ls –l >> MyFiles < Redirects input read Line < MyFiles ; echo $Line ( )Starts subshell in which to run command(s) A=5 ; echo $A ; (A=3;echo $A) ; echo $A Shell Metacharacters (wildcards)

20 Shell Metacharacters (Con’t) $ Substitutes variables $ Substitutes variables ?Matches for a single character ?Matches for a single character *Matches for zero or more characters *Matches for zero or more characters [abc] Matches for one character form a set of characters [abc] Matches for one character form a set of characters [!abc] Matches for one character not from a set of characters [!abc] Matches for one character not from a set of characters

21 Filename Generation.1 The shell will automatically generate a list of file names for you based on the use of wild card characters The shell will automatically generate a list of file names for you based on the use of wild card characters Characters: Characters: * any string of characters of any length * any string of characters of any length ? any single character ? any single character [aeiou] any single character from the set inside the brackets [aeiou] any single character from the set inside the brackets Can use with echo to find out what files match a pattern Can use with echo to find out what files match a pattern echo a* files starting with ‘a’ echo a* files starting with ‘a’ echo *[0-9]* files containing a digit echo *[0-9]* files containing a digit echo [A-Ka-k]?? Filenames 3 characters long that start with an upper- or lower-case a,b,c,d,…,k echo [A-Ka-k]?? Filenames 3 characters long that start with an upper- or lower-case a,b,c,d,…,k

EX: EX: A=‘ls’ ; echo $A A=‘ls’ ; echo $A A=`ls`; echo $A A=`ls`; echo $AA=`ls` for F in $A do do if [ -f $F ] ; then if [ -f $F ] ; then echo -n $F echo -n $F fi fi done done `back ticks`

23 Script.1 Write the following shell script Write the following shell script #!/bin/bash # Name this script quotes VAR="Hi There" CMD=date echo Trying with single quotes echo '$VAR `$CMD`' echo " " echo Trying with double quotes echo "$VAR `$CMD`"

24 I/O Redirection.1 As discussed earlier, there are three files automatically opened for each process: As discussed earlier, there are three files automatically opened for each process: I/O redirection is about “hooking” these files up to other files I/O redirection is about “hooking” these files up to other files When a file descriptor is assigned to something other than a terminal, it is called I/O redirection When a file descriptor is assigned to something other than a terminal, it is called I/O redirection Name File descriptor Purpose stdin 0 keyboard stdout 1 screen (normal output) stderr 2 screen (error messages)

25 File descriptor.1 A small unsigned integer A small unsigned integer Used by kernel to reference open file and I/O stream Used by kernel to reference open file and I/O stream The first three, 0, 1, and 2 are assigned to your terminal. The first three, 0, 1, and 2 are assigned to your terminal. A new open file will get a number of ? A new open file will get a number of ?

26 First three file descriptors.2 STDIN = 0/dev/stdin STDIN = 0/dev/stdin STDOUT=1 /dev/stdout STDOUT=1 /dev/stdout STDERR=2/dev/stderr STDERR=2/dev/stderr

27 We have already seen two types of redirection We have already seen two types of redirection echo hi there >output Output (stdout) goes to disk file instead of screen Output (stdout) goes to disk file instead of screen – cat file | tr 'a-z' 'A-Z' Output goes to stdin of tr instead of screen Output goes to stdin of tr instead of screen I/O Redirection.2 hi there cattr X X X echo

28 I/O Redirection.3 Appending to a file Appending to a file – cat file1 >>file2 Redirects output to file2, just as with > Redirects output to file2, just as with > HOWEVER … appends to file2 instead of overwriting HOWEVER … appends to file2 instead of overwriting Try this: Try this: echo "Today's time and date" >newfile date >>newfile cat newfile

29 I/O Redirection.4 Redirecting stderr Redirecting stderr – cat * >allfiles If some of the files are not readable, will cause errors to go to screen. If some of the files are not readable, will cause errors to go to screen. Solution, send stderr (file descriptor 2) to errorfile Solution, send stderr (file descriptor 2) to errorfile – cat * >allfiles 2>errorfile 01 2 cat allfiles errorfile

30 I/O Redirection.5 cat * >allfiles 2>errorfile If we NEVER want to see the errors we can send them to a special file called /dev/null: cat * > allfiles 2>/dev/null /dev/null - is the “bit bucket” It’s like a black hole from which no data ever returns 01 2 cat allfiles

31 Try and Learn The find command will walk the file system looking for files the match your criteria and then act on them. The find command will walk the file system looking for files the match your criteria and then act on them. – find / -name ′*.so’ –print Finds all files whose names end with ‘.so’ starting at the root (-name ’*.so’) Finds all files whose names end with ‘.so’ starting at the root (-name ’*.so’) When it finds one, it prints its name (-print) When it finds one, it prints its name (-print) Because you do not have permission to all directories, this will generate error messages Because you do not have permission to all directories, this will generate error messages Try it as written above and see Try it as written above and see Execute this command, discarding error messages Execute this command, discarding error messages

32 I/O Redirection.5 Sending stdout and stderr to same place Sending stdout and stderr to same place – cat * >bigfile 2>&1 The 2>&1 says to send file descriptor 2 to the same place as 1. The 2>&1 says to send file descriptor 2 to the same place as 1. Order counts … consider: cat * 2>&1 >bigfile Order counts … consider: cat * 2>&1 >bigfile Why? At the time we reach 2>&1, 1 is still connected to the screen Why? At the time we reach 2>&1, 1 is still connected to the screen 01 2 cat allfiles 01 2 cat allfiles

33 find Rerun the find command redirecting all of its output to a file Rerun the find command redirecting all of its output to a file Redirect both normal output and error output Redirect both normal output and error output Now, try this: Now, try this: – find / -name '*.so' -print >file1 2>&1 >file2 what happens above? Can you explain? what happens above? Can you explain?

34 I/O Redirection.6 We can also redirect stdin We can also redirect stdin tr ‘a-z’ ‘A-Z’ <myfile tr ‘a-z’ ‘A-Z’ <myfile Many Unix commands read from stdin by default Many Unix commands read from stdin by default Some allow files to be specified on the command line Some allow files to be specified on the command line Others require this form of redirection to read from a file Others require this form of redirection to read from a file 01 2 tr myfile X

35 I/O Redirection.7 The exec command The exec command – exec <file Opens the file as stdin for the script. Since open files are part of the environment, this is also stdin for all commands, unless explicitly changed Opens the file as stdin for the script. Since open files are part of the environment, this is also stdin for all commands, unless explicitly changed Similarly can redirect other streams Similarly can redirect other streams exec >file exec >file exec 3<file1# file descriptor 3 connected to file1 anycommand 0<&3# connect stdin to file descriptor 3(file1) exec 3<&-# close file descriptor 3 exec 3<file1# file descriptor 3 connected to file1 anycommand 0<&3# connect stdin to file descriptor 3(file1) exec 3<&-# close file descriptor 3 exec 5>outfile# file descriptor 5 connected to outfile command1# default I/O connections command2 2>&5# stderr goes to outfile command3 1>&5# stdout goes to outfile exec 5>&-# close outfile exec 5>outfile# file descriptor 5 connected to outfile command1# default I/O connections command2 2>&5# stderr goes to outfile command3 1>&5# stdout goes to outfile exec 5>&-# close outfile

36 Issuing this command will send all stdout (fd 1) to file temp Sends stdout (fd1) BACK to screen /dev/tty

37

38 Pipelines.1 Each command has three I/O files automatically associated with it. Each has a name and a number (called the file descriptor) Each command has three I/O files automatically associated with it. Each has a name and a number (called the file descriptor) The output of one command can be connected to the input of the next using a pipeline cat file | tr a-z A-Z The output of one command can be connected to the input of the next using a pipeline cat file | tr ' a-z ' ' A-Z ' Prints the file … the output goes to ‘tr’ filter which (in this case) translates all lower case characters to their upper case counterparts Prints the file … the output goes to ‘tr’ filter which (in this case) translates all lower case characters to their upper case counterparts Name File descriptor Purpose stdin 0 keyboard stdout 1 screen (normal output) stderr 2 screen (error messages)

39 NOTICE HOW STDOUT OF ONE PROCESS IS CONNECTED TO STDIN OF ANOTHER 103

40 Pipelines.2 Pipelines can be very long cat file | head -75 | tail -10 Pipelines can be very long cat file | head -75 | tail -10 Using ‘man’ learn about head and tail Using ‘man’ learn about head and tail Explain what the above pipeline does, assuming the file has 110 lines of text in it Explain what the above pipeline does, assuming the file has 110 lines of text in it Pipelines are neat because we don’t have to create a bunch of temporary files! Pipelines are neat because we don’t have to create a bunch of temporary files! $ cat file > file.1 $ head -75 file.1 > file.2 $ tail -10 file.2 $ rm file.1 file.2

41 Try and Learn Type the command: Type the command: Using the file ‘file1’ from the last exercise, try cat file1 echo $? You should see a ‘0’ printed Using the file ‘file1’ from the last exercise, try cat file1 echo $? You should see a ‘0’ printed Now, change the permissions so that you do not have read permission for ‘file1’ Now, change the permissions so that you do not have read permission for ‘file1’ Repeat the above test Repeat the above test What is the output this time What is the output this time

42 Positional Parameters.1 Shell scripts can receive parameters Shell scripts can receive parameters The shell parses the command that starts the script into words The shell parses the command that starts the script into words Each word, except the script name, is passed to the script as a ‘positional parameter’ Each word, except the script name, is passed to the script as a ‘positional parameter’ You can get a parameter if you know its position on the line You can get a parameter if you know its position on the line $1 = 1 st parameter, $2 = 2 nd parameter, … $9 = ninth parameter, ${10} = tenth parameter. $1 = 1 st parameter, $2 = 2 nd parameter, … $9 = ninth parameter, ${10} = tenth parameter. $10 IS $1 with a “0” appended to it $10 IS $1 with a “0” appended to it

43 Try and Learn Write a script called swap that prints out its first two parameters in reverse order Write a script called swap that prints out its first two parameters in reverse order e.g., swap a b prints e.g., swap a b prints swap Hello “you guys” prints swap Hello “you guys” prints The builtin variable $# tells you how many parameters there are The builtin variable $# tells you how many parameters there are Change swap so that it prints an error message and exits if there are not exactly 2 arguments Change swap so that it prints an error message and exits if there are not exactly 2 arguments Use conditional execution and grouping as in: [ $# -eq 1 ] || { echo Error message; exit 1; } Use conditional execution and grouping as in: [ $# -eq 1 ] || { echo Error message; exit 1; }

44 test command Arguments indicate what test to perform Arguments indicate what test to perform Sort of like a Boolean expression Sort of like a Boolean expression test returns 0 if the test succeeds, 1 if it does not test returns 0 if the test succeeds, 1 if it does not Alternative syntax to test  [ … ] Alternative syntax to test  [ … ] Example: Example: if test -d pathname then echo It IS a directory fi if test -d pathname then echo It IS a directory fi if [ -d pathname ] # SPACES ARE VERY IMPORTANT HERE! if [ -d pathname ] # SPACES ARE VERY IMPORTANT HERE! then echo It IS a directory fi Typeform more types of tests Type man test form more types of tests

45 If statements options if [ $VAR -eq $VAR2 ]; then …;fi #equal to if [ $VAR -eq $VAR2 ]; then …;fi #equal to if [ $V -ne $V2 ]; then …;fi #not equal to if [ $V -ne $V2 ]; then …;fi #not equal to if [ $V -gt $V2 ]; then …;fi # greater than if [ $V -gt $V2 ]; then …;fi # greater than if [ $V -lt $V2 ]; then …;fi # less than if [ $V -lt $V2 ]; then …;fi # less than if [ $V -ge $V2 ]…#greater than or equal to if [ $V -ge $V2 ]…#greater than or equal to if [ $V -le $V2 ]… # less than or equal to if [ $V -le $V2 ]… # less than or equal to

46 Positional Parameters.2 If you know how many parameters there are, you can get at all of them using the shift command If you know how many parameters there are, you can get at all of them using the shift command shiftDeletes $1 and shifts all the rest to the left by 1 place: $1  $2  $3  … shiftDeletes $1 and shifts all the rest to the left by 1 place: $1  $2  $3  … Repeated use gets to all parameters Repeated use gets to all parameters Special for loop Special for loop for VARNAME# executes 1 time/parameter do...# $VARNAME changes to next # parameter on each iteration done for VARNAME# executes 1 time/parameter do...# $VARNAME changes to next # parameter on each iteration done

47 Positional Parameters.3 In addition to $#, there are two other ‘special’ variables for use with positional parameters In addition to $#, there are two other ‘special’ variables for use with positional parameters $*list of all parameters $*list of all parameters of all parameters of all parameters These appear to be the same These appear to be the same They differ when quoted with double quotes They differ when quoted with double quotes

48 Try One More Modify the for loop to Modify the for loop to for NAME in "$*" do echo done for NAME in "$*" do echo done Create a script with the above in it and run it Create a script with the above in it and run it Be sure to include some quoted phrases as well as single words in your parameters Be sure to include some quoted phrases as well as single words in your parameters Now … change the for statement to for NAME in and try again Now … change the for statement to for NAME in and try again How do these to differ? How do these to differ? What if you leave the quotes off? What if you leave the quotes off? Which appears to operate the same as the for NAME (by itself)? Which appears to operate the same as the for NAME (by itself)?

49 ‘Here’ Documents.1 It reads in inline text for a program expecting input It reads in inline text for a program expecting input Data is read by using << redirection Data is read by using << redirection Reads data starting at the next line of the script up to a special ‘marker’ line Reads data starting at the next line of the script up to a special ‘marker’ line Example: Example: echo “Start” cat << EOF This is the help for my script. Usage: $0 arg1 EOF echo “Start” cat << EOF This is the help for my script. Usage: $0 arg1 EOF NOTE: the ‘marker’ is the EOF following the << NOTE: the ‘marker’ is the EOF following the << The marker must be at the START of the line to be recognized as the END of the ‘document’ The marker must be at the START of the line to be recognized as the END of the ‘document’ The marker can be ANY word (EOF is not special) The marker can be ANY word (EOF is not special)

50 ‘Here’ Document.2 So … So … Each line of the ‘document’ is evaluated by the shell Each line of the ‘document’ is evaluated by the shell Variables are substituted (and other meta characters are evaluated) Variables are substituted (and other meta characters are evaluated) To avoid this, if you wish, single quote the marker To avoid this, if you wish, single quote the marker command << ‘EODATA’... EODATA command << ‘EODATA’... EODATA NOW … the lines of the ‘document’ are treated as though they are single quoted (i.e., they are not evaluated by the shell) NOW … the lines of the ‘document’ are treated as though they are single quoted (i.e., they are not evaluated by the shell) NOTE: the marker at the end of the ‘document’ is NOT quoted NOTE: the marker at the end of the ‘document’ is NOT quoted

51 Commands ps bg fg jobs kill and & ps bg fg jobs kill and & sleep sleep date date chmod chmod cd cd ls touch mkdir rmdir mv   cp   rm   cat   cut paste   sort uniq