Intermediate shell scripting in BASH Research Computing Services course week, fall 2012 Lecturer: Andreas Buzh Skau Head Engineer, RCS buzh@usit.uio.no http://folk.uio.no/buzh/bash/

11. april 2011 Ny Powerpoint mal 2011 What is a shell script? The shell is a computer program that presents you with the command line interface used on linux/unix systems. A script is a list of commands that you ask the shell to perform, like a theater director giving a script to his cast of actors. An actor reads, interprets and executes the script according to the director's instructions. The shell does the same with your scripts.

chmod +x myscript.sh and then running the command: 11. april 2011 Ny Powerpoint mal 2011 cd /tmp echo "some text" > some-file ls -l /tmp/some-file ps auxw | grep flowers In addition to just listing commands sequentially like this, shells provide us with the ability to use variables, tests, loops and other methods found in common programming languages like C, Python, Fortran etc Bash scripts are plain text files which can be executed either by making the file executable: chmod +x myscript.sh and then running the command: ./myscript.sh or by passing the filename to bash itself like this: bash myscript.sh” If the script is chmod +x and in your $PATH, you can type it's name anywhere, and it will run.

Scripts are executed line by line 11. april 2011 Ny Powerpoint mal 2011 To make scripts in Linux, you should add a “hashbang” to the first line of the file. The # is called a hash, and the ! Is a “bang”. Then; the absolute path to the interpreter which will be used to execute the script: /bin/bash Scripts are executed line by line This is what a simple script looks like, in it's entirety: #!/bin/bash echo Hello World

When assigning variables, we simply write variable=value 11. april 2011 Ny Powerpoint mal 2011 Variables: When assigning variables, we simply write variable=value Note! There is no space before or after the = sign. To get the contents of the variable we precede the name of the variable with a dollar sign $variable Before the shell executes a line, it will expand $variable to the actual value contained within. This means that in this case, $variable is translated to “value” before execution.

mypath=/tmp/buzh/stuff myfilename=filename myfile=$mypath/$myfilename Ny Powerpoint mal 2011 11. april 2011 Variables mypath=/tmp/buzh/stuff myfilename=filename myfile=$mypath/$myfilename mytext='Howdy partner!' mynumber=23 myarray=(apple banana cucumber daffodil) mkdir -p $mypath echo “$mytext” > $myfile echo $mynumber >> $myfile mv $myfile $myfile.${myarray[3]} for ((i=0; i<=$mynumber; i++)); do echo $i; done Usage:

if <condition>; then <commands>; fi Ny Powerpoint mal 2011 11. april 2011 Tests One of the most useful things in scripting are tests. They make up much of the logic of our scripts, checking whether some condition is true or false and then executing the appropriate commands. if <condition>; then <commands>; fi if [ -e "$myfile" ] then echo $mytext else echo "$myfile does not exist!" fi

[ is actually a program called "test" [ is actually a program called "test". You can find it's documentation with the command "man test". In this example we use "-e" to check for the existence of a file. It can also do arithmetic tests, string comparisons etc. You can nest if/then tests, and don't forget about elif: if [ $mynumber = 23 ]; then echo $mytext elif [ $mynumber = 24 ]; then echo "hello, this can't be right?" if [ $(date +%A) = Sunday ]; then export mynumber=23 echo mynumber is now 23 again else echo Try again next sunday, $(date -d 'next sunday' “+%D %h”) fi echo "This would be executed if my number is neither 23 nor 24" The program 'date' used above is also a gem of usefulness. See 'man date'.

The “if” command above does the same thing! Ny Powerpoint mal 2011 11. april 2011 If the return value of a test condition is 0, the test is logically true. If it is non-zero, it's logically false. This is the typical UNIX way of determining success or failure of a command. Consider this: ls /unlikely && echo YEAH || echo NOPE if (ls /unlikely); then echo YEAH; else echo NOPE; fi If the path /unlikely exists, ls will exit with a return code 0. If it does not exist, it will return a non-zero value. As you might know, the && command will read the exit status of the previous command and then execute the next command if the exit status was 0, a logical “and”. Conversely || will only execute the next command if the exit status is not 0, a logical “or”. The “if” command above does the same thing!

The “for” loop: for <item> in <list>; do <commands>; done The first argument in this example (<item>) is the name of a variable, to which each value contained in the list you supplied will be assigned. You take an arbitrary word (“fruit”), and through these commands it is used as a variable. Example: for fruit in apple banana orange pear; do echo $fruit is a fruit! done Make lists of anything: for stuff in $(someprogram | grep things); do.. Work with lists in files: for line in $(cat /path/to/somefile); do..

You can also write loops more like in traditional C, using arithmetic expansion (more on that later). Example: for ((i=0; $i<4; i++)); do if [ $i = 0 ]; then echo "The first number is $i"; else echo "then comes $i.." if [ $i = 4 ]; then echo "...and that was all."; fi fi; done Note the way the variable i is adressed in the first line. i= is an assignment, $i is a reference to the value contained in the variable and lastly, because we are using (( double parenthesis )) we can increase the value of the integer i with 1 by writing i++ Same stuff, different wrappings: for i in {0..4}; do.... for i in 0 1 2 3 4; do.... for i in $(seq 0 4); do....

Syntax: while <condition>; do <commands>; done Example: The “while” loop: Syntax: while <condition>; do <commands>; done Example: myvar=0 mylimit=10 while (( myvar < mylimit )) do echo "My Variable is $myvar, let's add to it" (( myvar += 1 )) done echo "This will echo only after the while loop has finished"

Functions: Functions in bash are much like functions in any other programming language.You declare it, then call it with the parameters you want Note that since bash reads the scripts line by line from the top, you must declare any functions before you use them - or else bash won't recognize the command when you try to call it! myfunction () { echo You just called my function } # To invoke this function, we call it as it were any normal command: myfunction # Or assign the output (“You just called my function”) to a var mytext=$(myfunction)

11. april 2011 Ny Powerpoint mal 2011 We've now seen some basic building blocks of a bash shell script. Next, let's see take a step back and look at how bash actually interprets our commands and scripts. echo Hello, $USER, how are you today? After bash reads this line, it's "expanded" before execution. So before the command is actually executed, it is rewritten, or expanded, like this: echo Hello, buzh, how are you today? ..if your username is buzh. The command is executed after this expansion has taken place. It's bash that does the parsing of the variables, the program "echo" has no knowledge of the variable $USER, only it's value To inspect what will actually be executed, the “echo” command is a good tool: echo echo Hello, $USER, how are you today?

In the above examples we use the $ Expansion also happens for other input with special meaning, not just variables. echo “Hello, $USER. 2 times 4 equals $(echo “2*4” | bc)” echo This host has $(grep -c bash /etc/passwd) bash users Bash will try to execute all the little sub-tasks first, then fill in the main command with the results, before executing the expanded command. In the above examples we use the $ sign to expand the result of another command. The paranthesis around the bc and grep commands above starts a “subshell”, another instance of bash, and the prepending $ means that the output of this is treated much like you would treat a variable.

(Some) Types of expansion: 11. april 2011 Ny Powerpoint mal 2011 (Some) Types of expansion: Curly brace: for n in {0..10}; do echo I can count to $n; done → for n in 0 1 2 3 4 5 6 7 8 9 10; do.... Command substitution: mycomputer="$(cat /etc/hostname)" → mycomputer=abel.uio.no Arithmetic expansion: Fifteen=$(( (10 / 2) + 10 )) → Fifteen=15 Filename expansion: echo /etc/pa* → /etc/pam.conf /etc/pam.d /etc/papersize /etc/passwd /etc/passwd-

Special Variables: $0 - the name of the parent, eg "scriptname" or "bash" $1 to $n - the parameter given on the n'th position “$*” - all the parameters passed as a string - REMEMBER TO QUOTE THIS “$@” - same as $*, but each word quoted - must also be quoted! $# the number of parameters passed, e.g. n from above safari () { if [ $# = 3 ]; then # we got 3 parameters echo First I saw a $1 echo Then I saw a $2 echo Finally a wild $3 appeared else # the if condition was false, so $# must be something other than 3 echo "You failed to pass three animals to the safari function" exit 1 fi } We invoke this like so: safari cow sheep gnu These variables work for scripts too, not just functions!

Further reading: http://www. tldp. org/LDP/abs/html/abs-guide Further reading: http://www.tldp.org/LDP/abs/html/abs-guide.html Advanced Bash Scripting Guide – This is a great reference, littered with examples and pretty much the definitive guide to bash scripting These slides: http://folk.uio.no/buzh/bash ..also a more wordy txt/html version of this course (a.k.a. the unreadable slides from last time) The most important thing: Study your repetitive tasks, and try to automate them!

Debugging and troubleshooting #!/bin/bash myletter=x echo Hello $USER echo In /tmp there are these files with the letter "$myletter" in them: echo $(ls /tmp | grep $myletter) buzh@stridselg ~ $ bash test.sh Hello buzh In /tmp there are these files with the letter x in them: xmms_buzh.0 buzh@stridselg ~ $ bash -x test.sh + myletter=x + echo Hello buzh + echo In /tmp there are these files with the letter x in them: ++ ls /tmp ++ grep x + echo xmms_buzh.0

This outputs: Hello Goodbye #!/bin/bash func () { myvar=$(echo $1 | sed s/Hello/Goodbye/) echo $myvar } greeting=Hello echo $greeting greeting=$(func $greeting) This outputs: Hello Goodbye

if (ping -c2 vin.uio.no &> /dev/null); then 11. april 2011 Ny Powerpoint mal 2011 while true; do if (ping -c2 vin.uio.no &> /dev/null); then echo We are online at $(date) else echo Uh.. seems down at $(date); fi sleep 30; done RESULTS=~/myresults for file in /mydata/*.dat; do TODAY=$(date +%d%m%y) # %d = day, %m month and so on. “date” is very handy! INFILE=$(basename $file) # Basename is a program that parses full file paths and # returns the filename only. Try it on $0 OUTFILE=$RESULTS/$file.$TODAY.log # Because of expansion, OUTFILE will become one long text string # e.g. /mydata/sampledata12345.dat.120412.log mycoolprogram $INFILE &> $OUTFILE done