CSc 352 Shell Scripts Saumya Debray Dept. of Computer Science The University of Arizona, Tucson debray@cs.arizona.edu

What is a shell script? A shell script is a list of commands to be run by a shell basically a program, but using shell commands instead of C or Java statements Why? automate repetitious tasks e.g.: testing a program on a large set of test inputs package up commonly executed command sequences create our own commands

Creating and executing a shell script Create a file, say “foo”, containing the commands to be executed by the script Execute the script: invoke the appropriate shell with foo as argument, e.g.: bash foo csh foo chmod a+x foo ./foo

A simple example a shell script to print “hello world!” the “echo” command, as executed from the command line contents of the shell script file “hello_world.script” this script behaves the same for a variety of different shells with appropriate permissions, the script can be executed directly

Syntax issues across shells Different shells have slightly different syntax rules the same script may not work on all shells: script contents different behaviors

Which shell runs my script? my current shell is csh the script executes as expected when csh is invoked explicitly if not invoked explicitly, the default shell /bin/sh is used

Solution: specify shell in the script #!pathToShell on first line specifies which shell to use executes as expected when invoked in csh executes as expected when invoked in bash

Specifying the shell interpreter #!/bin/csh ‘#!’ at the beginning of a file indicates that it is a script The rest of the first line specifies (absolute) path to the interpreter What if the interpreter is at a different location on a different system? use /usr/bin/env to improve portability: #!/usr/bin/env csh

Variables Variables different than in C or Java: don’t have to be declared in advance they are untyped: the same variable can hold an integer value or a string Syntax for using variables (bash): Defining the value of a variable X: X=value Using the variable X: $X or ${X}

Example variable definitions variable uses undefined variable evaluates to empty string

Guarding against uninitialized variables -u : “exit if using an unitialized variable” -u : works if we specify /bin/bash but not with /usr/bin/env

Guarding against uninitialized variables

Bash scripts: whitespace without whitespace  the command works as expected with whitespace  weird error message variable value not set as expected

Different kinds of quotes back quotes: evaluates to the output of the enclosed commands single quotes: variables not expanded double quotes: variables inside the quotes are expanded

A simple scripting problem Iterate over the files in a given directory: for each file, express its size in human-readable form (i.e., using KB and MB where appropriate): give 2 decimal places for MB sizes; 1 decimal place for KB sizes functionality similar to using “ls –lh” Requires: list files in a directory [valid directory?  if-then-else] iterate over these files obtain the size of each file perform arithmetic [KB/MB decision  if-then-else]

The test command Used to guide control flow in conditionals and loops Syntax: test operand1 operator operand2 test operand often abbreviated to: [ operand1 operator operand2 ] [ operand ] see “man test” for a list of operators Returns a nonzero status if the test is true; zero o/w

Example: test

Example: if-then-else if-then-else command

Example: if-then-else need a ‘;’ between the test expression and ‘then’

Example: if-then-else

Alternative formulation of test

For Loops Syntax 1: for X in word ; do cmds done Convenient for iterating over lists Syntax 2: for (( e1; e2; e3 )) ; do cmds done Convenient for iteration involving arithmetic expressions

For Loops: Example 1

For Loops: Example 2

Iterating over lists: pitfalls ! print out the first couple of lines of a file listing

Iterating over lists use quotes to prevent splitting at whitespace Note: the loop iterates only once – not once per file!

Arithmetic arithmetic evaluation doesn’t happen automatically; must be specified explicitly get file size (see “man stat” for other file properties)

Arithmetic integer arithmetic only! arithmetic evaluation: $(( expr ))

get bc to do the floating point math for us Arithmetic get bc to do the floating point math for us

Command-line arguments Arguments at known positions, e.g., 1st, 2nd, ..., can be accessed using $1, $2, … $0 = the name of the script file Total no. of arguments: $# However, we can’t get to the ith argument this way $i  value of variable i

Example

Iterating over the argument list

Getting at the entire list of arguments

Iterating over the argument list Placement of quotes matters: all of the arguments are seen as a single word

Iterating over the argument list Placement of quotes matters:

Arguments with whitespace $@ is similar to $*, but each argument is a quoted string no interpretation or expansion allows us to properly access arguments that may contain whitespace needs to be quoted “…”

Arguments with whitespace

Debugging

prints shell input lines as they are read Debugging: -v prints shell input lines as they are read

Debugging: -v these are the lines from the script that are being executed Note that variable values etc. are not being shown

Debugging: -x prints out command traces before they are executed

Debugging: -x variable values, command arguments are printed out

Debugging: summary Debugging can be done by observing what commends get executed “set –v” prints out shell commands as they are read in does not print out runtime variable values “set –x” prints out a trace of the command before it is actually executed prints out variable and argument values Debugging can be turned off using “set +v”, “set +x” useful for localizing which parts of the script are observed

String manipulation shell scripts often involve string manipulation e.g., to pattern match on file names, create new files based on existing files, etc. bash gives extensive facilities for string matching and manipulation see “man expr”

A sample problem Write a bash script to convert Unix make files to “mymake” make files unix make mymake a : b.o c.o d.o gcc *.o –o a b.o : b.c b.h gcc –Wall –c b.c @target a : b.o c.o d.o @cmd gcc *.o –o a @target b.o : b.c b.h @cmd gcc –Wall –c b.c 

Attempt 1 Ignore definitions Assume Issues: assume make file does not have stuff like “CC = gcc” Assume user has added whitespace where necessary e.g.: on either side of ‘ : ’ in rules all dependencies have been made explicit Issues: need to process a line at a time lines may contain whitespace need to check whether a line starts with a ‘\t’

Attempt 1

Attempt 1 IFS = “Internal Field Separator” used by bash to split the input into words the new definition of IFS means that the input is now split at newlines identify lines that begin with a tab

Attempt 1

Attempt 2 relax assumption about whitespace around : Issue: want: input: “a.o: a.c b.h c.h” output: “@target a.o : a.c b.h c.h” Issue: replace ‘:’ by ‘ : ’ bash offers two replacement operators: ${str/substr/replacement} replaces the first occurrence of substr in str with replacement ${str//substr/replacement} replaces all occurrences of substr in str with replacement

Attempt 2

Attempt 2

Only the first ‘:’ had whitespace inserted Attempt 2 Only the first ‘:’ had whitespace inserted

“replace all occurrences” Attempt 2 “replace all occurrences”

Attempt 2

Attempt 3 Relax assumption about definitions Issues: allow “CC=gcc”, “CFLAGS= –Wall –g –c” assume no whitespace around ‘=‘ Issues: identify definitions, i.e., lines that contain a ‘=‘ extract LHS, RHS of the definition replace every occurrence of LHS by RHS

Attempt 3 if the line contains a ‘=‘ … figure out where it occurs LHS = substring from position 0 to position of ‘=‘ RHS = substring after position of ‘=‘ to end substitute RHS for every occurrence of LHS

Attempt 3

Attempt 3 Oops!

Attempt 3

Attempt 3