1. CIT 470: Advanced Network and System Administration
Shell Scripting Techniques
CIT 470: Advanced Network and System Administration

2. CIT 470: Advanced Network and System Administration
Topics
Shell Variables
Command Arguments
Executing Commands
I/O Redirection
Loops
Functions
Debugging Scripts
CIT 470: Advanced Network and System Administration

3.
Variables

4. Setting Shell variables
Assign a value to a variable:
varname=value
Examples
monty=python
spam=“spam, spam, spam, spam”
PATH=/bin:/usr/ucb:/usr/bin
Notes
No spaces on either side of equal sign.
CIT 140: Introduction to IT

5. CIT 140: Introduction to IT
Using Shell Variables
> spam=eggs
> echo $spam
eggs
> echo spam
spam
> echo \$spam
$spam
> spam=spam and eggs
bash: and: command not found
> spam="spam and eggs"
spam and eggs
> spam=c*
cit140 csc382 csc501
CIT 140: Introduction to IT

6. CIT 140: Introduction to IT
Command Substitution
Command Substitution: When a command is enclosed in back quotes, the shell executes the command and substitutes the command (including back quotes) with the output of the command. Two different syntaxes exist for substitution:
`command`
$(command)
CIT 140: Introduction to IT

7. Using Command Substitution
> dir=`pwd`
> echo $dir
/home/b/waldenj
> echo "The current directory is $dir"
The current directory is /home/b/waldenj
> echo "The current date and time is `date`"
The current date and time is Fri Feb 25 15:47:16 EST 2011
CIT 140: Introduction to IT

8. CIT 140: Introduction to IT
Exporting Variables
export [name-list]
Purpose: Export the names and copies of the current values in the ‘name-list’ to every command executed from this point on.
Example:
> grep PATH .bashrc
PATH=/bin:/usr/bin:/usr/local/bin:/usr/ucb
MANPATH=/usr/local/man:/usr/man:/usr/X11R6/man
export PATH MANPATH
CIT 140: Introduction to IT

9. CIT 140: Introduction to IT
Resetting Variables
unset [name-list]
Purpose Reset or remove the variable or function corresponding to the names in ‘name-list’, where ‘name-list’ is a list of names separated by spaces.
> food1=spam
> food2=eggs
> echo "I like $food1 and $food2"
I like spam and eggs
> unset food1 food2
I like and
CIT 140: Introduction to IT

10. Reading from Standard Input
read variable-list
Purpose: Read one line from standard input and assign words in the line to variables in ‘name-list’.
CIT 140: Introduction to IT

11.
Arguments

12. Shell Script Arguments
$ cat cmdargs_demo
#!/bin/sh
echo “The command name is: $0.”
echo “The number of command line arguments passed as parameters are $#.”
echo “The value of the command line arguments are: $1 $2 $3 $4 $5 $6 $7 $8 $9.”
echo “Another way to display values of all of the arguments:
echo “Yet another way is: $*.”
exit 0
$ cmdargs_demo a b c d e f g h i
The command name is: cmdargs_demo.
The number of command line arguments passed as parameters are 9.
The value of the command line arguments are: a b c d e f g h i.
Another way to display values of all of the arguments: a b c d e f g h i.
Yet another way is: a b c d e f g h i.
$ cmdargs_demo One Two 3 Four 5 6
The number of command line arguments passed as parameters are 6.
The value of the command line arguments are: One Two 3 Four 5 6 .
Another way to display values of all of the arguments: One Two 3 Four 5 6.
Yet another way is: One Two 3 Four 5 6.
CIT 140: Introduction to IT

13. Looping Over Arguments
#!/bin/sh # # Set file ownerships to root and set permissions for filename in $* do echo "Setting ownerships and permissions for $filename" chown root $filename chmod 0750 $filename done
CIT 470: Advanced Network and System Administration

14. Looping Over Arguments
> sudo ./set_perms_owns *txt
Setting ownerships and permissions for 1.txt
Setting ownerships and permissions for 2.txt
Setting ownerships and permissions for 3.txt
> ll
total 4
-rwxr-x--- 1 root jw 0 Feb 25 16:38 1.txt*
-rwxr-x--- 1 root jw 0 Feb 25 16:38 2.txt*
-rwxr-x--- 1 root jw 0 Feb 25 16:38 3.txt*
-rwxr-xr-x 1 jw jw 194 Feb 25 16:38 set_perms_owns*
CIT 470: Advanced Network and System Administration

15. Checking Number of Arguments
#!/bin/sh if [ $# -lt 3 ]; then echo "Only $# arguments given; need exactly 3 arguments." exit 1 elif [ $# -gt 3 ]; then echo "Too many arguments; need exactly 3 arguments." else echo "Argument count correct. Proceeding..." fi
CIT 470: Advanced Network and System Administration

16.
Commands

17. CIT 470: Advanced Network and System Administration
Success or Failure?
> mkdir /tmp/a > echo $? 0 > mkdir /tmp/b/c mkdir: cannot create directory `/tmp/b/c': No such file or directory 1
CIT 470: Advanced Network and System Administration

18. CIT 470: Advanced Network and System Administration
Checking Success
#!/bin/sh
mkdir $1
if [ $? -eq 0 ]; then
echo success
else
echo failure fi
CIT 470: Advanced Network and System Administration

19. Safely Modifying Files
#!/bin/bash CFGDIR=/etc/openldap CFGFILE=ldap.conf # If CFGDIR is missing, create it if [ ! -d $CFGDIR ]; then mkdir -p $CFGDIR fi # Save original version of CFGFILE before modifying if [ -f $CFGFILE ]; then mv $CFGFILE $CFGFILE.orig
CIT 470: Advanced Network and System Administration

20.
I/O Redirection

21. CIT 470: Advanced Network and System Administration
Eliminating Output
Shell scripts should not print all of the output from all the commands run in the scripts.
Eliminating STDOUT
command >/dev/null
Eliminating STDERR
command 2>/dev/null
Eliminating both STDOUT and STDERR
command >/dev/null 2>&1
CIT 470: Advanced Network and System Administration

22. CIT 470: Advanced Network and System Administration
Logging Output
Shell scripts may save command output to a log file to aid in debugging.
Logging STDOUT
command >>script.log
Logging STDERR
command 2>>script.log
Logging both STDOUT and STDERR
command >>script.log 2>&1
CIT 470: Advanced Network and System Administration

23. Storing Command Output
> f=$(find . -name '*.txt')
find: ./a: Permission denied
> echo $f
./3.txt ./2.txt ./1.txt
> f=$(find . -name '*.txt' 2>&1)
find: ./a: Permission denied ./3.txt ./2.txt ./1.txt
CIT 470: Advanced Network and System Administration

24.
Loops

25. CIT 470: Advanced Network and System Administration
The for loop
kosh> for i in > do > echo $i > done
CIT 470: Advanced Network and System Administration

26. Processing files with for
# Rename files with numbers 1 through 5 for i in do mv myfile-$i.txt file-$i.txt.bak done # Rename all files in directory with .bak extension for file in * mv $file $file.bak
CIT 470: Advanced Network and System Administration

27. Processing files with for
# Rename all .txt files in directory with .bak extension for file in *.txt do mv $file $file.bak done # Replace servername placeholder with “cit470-team1” in all # configuration files in this directory, saving old .conf files # with .orig extension for file in *.conf do ruby –i.orig -pe 'gsub(/servername/, “cit470-team1")' $file
CIT 470: Advanced Network and System Administration

28.
Parsing Files

29. CIT 140: Introduction to IT
Reading Files
The read command can use I/O redirection.
> read line </etc/passwd
> echo $line
root:x:0:1:Super-User:/:/sbin/sh
CIT 140: Introduction to IT

30. CIT 140: Introduction to IT
Reading Files
Loops can use I/O redirection with read.
while read line do
echo $line
done </etc/passwd
CIT 140: Introduction to IT

31. CIT 140: Introduction to IT
Example: cat
#!/bin/sh
if [ $# -lt 1 ]; then
echo "Usage: cat file1 [file2, ...]" fi
for file in do
while read line
echo $line
done <$file
done
CIT 140: Introduction to IT

32. Parsing files using read
#!/bin/sh
TMPFILE=hosts.tmp
egrep -v "^#" /etc/hosts >$TMPFILE
while read ipaddress hostname ignore do
echo "Host $hostname has IP address $ipaddress."
done <$TMPFILE
rm $TMPFILE
> ./hosts.sh
Host localhost has IP address
Host zappa has IP address
Host mailfe2 has IP address
Host axp1 has IP address
CIT 140: Introduction to IT

33. CIT 470: Advanced Network and System Administration
Ruby Options
-e string This option executes the string as a ruby program. -n This option runs the program as if it were enclosed in the following loop: while gets # Read a line of input into $_ # Program text here end -p This option runs the program as if it were written in the following loop: print # Output $_
CIT 470: Advanced Network and System Administration

34. CIT 470: Advanced Network and System Administration
Substitutions
# substitute (find and replace) "foo" with "bar" on each line $ ruby -pe 'gsub(/foo/, "bar")' < file.txt # substitute "foo" with "bar" ONLY for lines containing "baz" $ ruby -pe 'gsub(/foo/, "bar") if $_ =~ /baz/' < file.txt # substitute "foo" w/ "bar" EXCEPT for lines containing "baz" $ ruby -pe 'gsub(/foo/, "bar") unless $_ =~ /baz/' < file.txt
CIT 470: Advanced Network and System Administration

35. CIT 470: Advanced Network and System Administration
Whitespace Removal
# delete leading whitespace from beginning of each line $ ruby -pe 'gsub(/^\s+/, "")' < file.txt # delete trailing whitespace from end of each line $ ruby -pe 'gsub(/\s+$/, $/)' < file.txt # delete BOTH leading and trailing whitespace from each line $ ruby -pe 'gsub(/^\s+/, "").gsub(/\s+$/, $/)' < file.txt
CIT 470: Advanced Network and System Administration

36. Deleting Specified Lines
# print all of file except btw 2 regular expressions, /foo/ & /bar/
$ ruby -ne = true if $_ =~ /foo/; puts = false if $_ =~ /bar/' < file.txt
# print file except for first 10 lines
$ ruby -pe 'next if $. <= 10' < file.txt
# print file except for every 8th line
$ ruby -pe 'next if $. % 8 == 0' < file.txt
# print file except for blank lines
$ ruby -pe 'next if $_ =~ /^\s*$/' < file.txt
CIT 470: Advanced Network and System Administration

37. CIT 470: Advanced Network and System Administration
Editing in Place
-i extension Specifies in-place-edit mode. The extension, if specified, is added to old file name to make a backup copy. For example: > echo matz > /tmp/junk > cat /tmp/junk matz > ruby -p -i.bak -e '$_.upcase!' /tmp/junk MATZ > cat /tmp/junk.bak
CIT 470: Advanced Network and System Administration

38. CIT 470: Advanced Network and System Administration
Functions
CIT 470: Advanced Network and System Administration

39. CIT 140: Introduction to IT
Functions
function () { commands; }
“Mini-scripts” inside your script
Invoked like another shell script.
Handle arguments like shell scripts do.
Exit status is exit status of last command.
CIT 140: Introduction to IT

40. CIT 140: Introduction to IT
Function Example
#!/bin/sh
findstr() {
dir=$1
string=$2
find $dir -type f -print | xargs fgrep $string }
if [ $# -ne 2 ]; then
echo "Usage: findstring dir string"
exit 1 fi
findstr $1 $2
CIT 140: Introduction to IT

41. Function Reuse Example
#!/bin/sh
backupfile() {
target=$1
if [ -f $target ]; then
cp -p $target $target.bak fi }
if [ $# -lt 1 ]; then
echo "Usage: backupdir dir"
exit 1
cd $1
for file in * do
backupfile $file
done
CIT 140: Introduction to IT

42. CIT 140: Introduction to IT
Why use Functions?
Organize your program.
Break up program into pieces that fit on one screen so you can read and understand it.
Code reuse
Can call a function multiple times.
DRY (Don’t Repeat Yourself) Principle.
CIT 140: Introduction to IT

43.
Debugging

44. Debugging Shell Programs
Noexec: check errors but don’t run
> sh –n script
set –o noexec
Verbose trace (can be combined with –n)
> sh –v script
set –o verbose
Short trace
> sh –x script
set –o xtrace
CIT 140: Introduction to IT

45. Debugging Shell Scripts
> sh –n backupdir.sh tmp
> sh -x backupdir.sh tmp
+ [ 1 -lt 1 ]
+ cd tmp
+ backupfile crypt.txt
target=crypt.txt
+ [ -f crypt.txt ]
+ cp -p crypt.txt crypt.txt.bak
+ backupfile lcdays.txt
target=lcdays.txt
+ [ -f lcdays.txt ]
+ cp -p lcdays.txt lcdays.txt.bak
+ backupfile who.txt
target=who.txt
+ [ -f who.txt ]
+ cp -p who.txt who.txt.bak
CIT 140: Introduction to IT

46. CIT 470: Advanced Network and System Administration
Key Points
Always check for correct arguments.
Always print a usage message.
Always put a comment block at top with
Names, team, instructor, class, assignment #
Description of what the script does
Always save original files before editing.
CIT 470: Advanced Network and System Administration

47. CIT 470: Advanced Network and System Administration
References
Carl Abling, JP Vossen, Cameron Newham, bash Cookbook, O’Reilly, 2007.
Bruce Blinn, Portable Shell Programming, Prentice Hall PTR, 1996.
Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly, 2002.
Thomas A. Limoncelli and Christine Hogan, The Practice of System and Network Administration, Addison-Wesley, 2002.
Evi Nemeth et al, UNIX System Administration Handbook, 4th edition, Prentice Hall, 2010.
Arnold Robbins and Nelson H.F. Beebe, Classic Shell Scripting, O’Reilly, 2005.
CIT 470: Advanced Network and System Administration