1. CS252: Systems Programming Ninghui Li Slides by Prof. Gustavo Rodriguez-Rivera Topic 7: Unix Tools and Shell Scripts

2. Common UNIX Commands

3. Basic UNIX Commands There are many UNIX commands that can be very useful They output of one can be connected to the input of the other using PIPES They are part of the UNIX distribution. Open source implementations exist and they are available in Linux.

4. ls – list files and directories ls file list Examples ls –al - Lists all files including hidden files (files that start with “.”) and timestamps. ls –R dir - Lists recursively all directories and subdirectories in dir.

5. cp – Copy files cp file1 file2 cp file1 file2 … destdir Copies one or more files to a destination. Examples: cp a.txt dir1 Copies file a.txt into dir1 cp a.txt b.txt Create a copy of a.txt called b.txt cp –R dir1 dir2 Copy recursively directories and subdirectories of dir1 into dir2.

6. mv – Move a file mv file1 destdir Moves file1 to destdir Examples: mv a.txt dir1 Move file a.txt into directory dir1 mv a.txt b.txt Rename file a.txt into b.txt

7. rm – Remove a file rm file1 file2 … Removes a list of files Examples: rm a.txt b.txt Remove files a.txt and b.txt rm –f a.txt Remove a.txt. Do not print error message if fails. rm –R dir1 Remove dir1 and all its contents.

8. mkdir – Make a directory mkdir dir1 … Examples mkdir dir1 Create directory dir1 mkdir –p dir1/dir2/dir3 Make parent directory an subdirectories if it they do not exist.

9. rmdir – Remove a directory rmdir dir1 … Examples rmdir dir1 Delete directory dir1 if it is empty rmdir –rf dir1 Delete dir1 and its contents, be careful when using it -r means resursive -f mean forces the delete (even when nonempty).

10. Show file content cat [file … ] Concatenate the contents of one or more files, and show content on stdout more file less file Show content of a file one page at a time

11. head – Lists the first few lines head file List the first 10 lines of a file Example: head myprog.c Lists the first 10 lines of myprog.c head -5 myprog.c List the first 5 lines of myprog.

12. tail – Lists the last few lines tail file List the last 10 lines of a file Example: tail myprog.c list the last 10 lines of myprog.c tail -3 myprog.c list the last 3 lines of myprog.c tail –f a.log It will periodically print the lines of a.log as they are added.

13. man – Print manual pages man command Print the manual page related to command. Examples: man cp Print the manual pages related to copy man –k pthread Print all manual pages that contain string “pthread”. man –s 3 exec Print manual page of exec from section 3 Man Pages are divided into sections: Section 1 – UNIX commands ( E.g. cp, mv etc) Section 2 – System Calls (E.g. fork) Section 3c – C Standard Library Example “man –s 1 printf” and “man –s 3c printf” give different man page. One printf from the shell, and the other from the lib C library.

14. grep – Find lines grep pattern file1 file2 … Print lines that contain “pattern” Examples: grep hello a.txt Print the lines in a.txt that contain hello grep hello * */* */*/* Print the lines that contain hello in any file in the directory and subdirectories.

15. whereis – Where a file is located Where file Prints the path of where a file is located. It only works if the OS created a database with the files in the file system. Example bash-4.1$ whereis apachectl /p/apache/apachectl (PATH=/p/apache) /p/apache-php/bin/apachectl (PATH=/p/apache-php/bin) /p/apache/man/man8/apachectl.8 (MANPATH=/p/apache/man) /p/apache-php/man/man8/apachectl.8 (MANPATH=/p/apache-php/man)

16. which – Path of a command which command Prints the path of the command that will be executed by the shell if “command” is typed. Example : bash-4.1$ which ps /usr/ucb/ps bash-4.1$ whereis ps /usr/bin/ps (PATH=/usr/bin) /usr/ucb/ps (PATH=/usr/ucb) bash-4.1$ export PATH=/usr/bin:$PATH bash-4.1$ which ps /usr/bin/ps

17. find – Execute a command for a group of files find – Recursively descends the directory hierarchy for each path seeking files that match a Boolean expression. Examples: find. -name "*.conf" -print Find recursively all files with.conf extension. find. -name “*.conf" -exec chmod o+r '{}' \; Find recursively and make readable by others all files with.conf extension

18. awk – pattern scanning and processing language An awk is a txt processing program. The program file has a sequence of rules of the form: pattern {action} … default {action} Where pattern is a regular expression. And action is a sequence of statements that run when the pattern is matched with the text. Optional

19. Awk examples You can also use awk for simple text manipulation: Example: awk '{print $1}' m.txt print the first word of each line in file m.txt Optional

20. sed – A simple line editor Used for simple text processing Example: sed s/current/new/g hello.sh > hello2.sh Replaces all the instances of current to new and redirects the output to hello2.sh Optional

21. Shell Scripting

22. Shell Programs Shells are programs used to interact with a computer using a command line. They used to be the only way that users could interact with the computer. Now there are Graphical Shells like Gnome, KDE, Windows Explorer, Macintosh Finder, etc that offer similar functionality. GUI Shells can do many of the tasks of command line shells but not all of them. Shells are used to make many administration procedures automatic: backup, program installation, Web services.

23. Shell Programs /bin/sh – Standard UNIX Shell /bin/ksh – Korn shell (more powerful) /bin/bash – GNU shell /bin/tcsh – Some line editing. Bash is becoming widely available in the UNIX word since it is the standard Linux shell.

24. Shell Scripts One can create shell scripts to do particular tasks, which include multiple lines of commands. A shell script file can be interpreted by a shell

25. Shell Scripts When a command is typed, the OS checks first if the file is an executable file with a known format (ELF, a.out). If it is not, then it checks the first line for a “#!interpreter-program”, if it is there, then it runs the interpreter program and passes the file as input.

26. hello.sh - Example of a Shell Script #!/bin/bash # # This shell script prints some data about the user # and shows how to use environment variables # echo "Hello $USER" echo "Welcome to CS252" echo "Your home directory is \"$HOME\"" echo "And your current directory is \"$PWD\""

27. Bash: Shell Expansions Brace expansion {}: E.g., echo a{b,c,d}e Tilde expansion ~: home directory E.g., cd ~, cd ~cs252 Variable expansion $: Environment var, or other var in a script Command substitution $(command) or `command` Replace result of command execution there

28. files.sh - Example of a Shell Script #!/bin/bash # This print the file names in current directory # twice for file in $(ls) do echo $file done for file in `ls` do echo $file done

29. Bash: Quoting Single quotes Preserves the literal value of a string E.g., echo ‘$PATH’ shows $PATH Double quotes, preserves literal value, except $NAME is replaced with the value of the environment var (empty string if undefined) E.g., echo “$PATH ls” Command substitution: E.g., echo “$(ls)”; echo “`ls`” Escape: \$, \`, \ , \\ replaced with $, `, , \ respectively

30. Bash: List of Commands A sequence of commands separated by ‘;’, ‘&’, ‘&&’, ‘||’, and optionally terminated by ‘;’, ‘&’, or newline & causes a command to be executed in the background, i.e., a new process is started to run it, and the shell does not wait for it to complete Command “fg” brings the background process into foreground ;causes a command to be executed sequentially In command1 && command2 We execute command2 if and only if command1 returns an exit status of zero In command1 || command2 We execute command2 if and only if command1 returns a non-zero exit status

31. Bash Conditionals if test-commands; then consequent-commands; [elif more-test-commands; then more-consequent-commands;] [else alternate-commands;] fi The test-commands list is executed, and if its return status is zero, the “then” branch is executed. (Similar to ifz)

32. Bash: Test Commands Either “test expression”, or “[expression]” or “[[ expression ]]” Return a status of 0 or 1 depending on the evaluation of the conditional expression expression There are differences between […] and [[…]]; however, we do not care. Understanding the following examples suffices for this course: String comparison: =, !=[$a=a] [$a != b] Integer comparison: -gt, -lt, -ge, -le, -eq, -ne, File check: -d fileTrue if file exists and is a directory. -e fileTrue if file exists. -f fileTrue if file exists and is a regular file. See http://mywiki.wooledge.org/BashFAQ/031

33. Bash Loop for name in [words …]; do commands; done Expand words, and execute commands once for each member in the resultant list, with name bound to the current member. while test-commands; do consequent-commands; done Execute consequent-commands as long as test-commands has an exit status of zero. The return status is the exit status of the last command executed in consequent-commands, or zero if none was executed.

34. Bash: Special Variables Individual arguments: $1, $2, … Number of arguments: $# All Arguments $@ or $* Script Name: $0 Process ID: $$ Process id of the shell Exit Code: $? Exit code of the last command executed

35. hello-loop.sh – Another example #!/bin/bash # # This shell script prints hello to all the friends you # pass as parameter if [ $# -lt 1 ] then echo echo "$0 needs at least one argument" echo " Eg." echo " $0 Mickey Donald Daisy" fi for friend in $* do echo "Hello $friend" done

36. mail-hello.sh - Send e-mail #!/bin/sh # # This script builds a simple message and mails it to # yourself # echo "Hello $USER!!" > tmp-message echo >> tmp-message echo "Today is" `date` >> tmp-message echo >> tmp-message echo "Sincerely," >> tmp-message echo " Myself" >> tmp-message /usr/bin/mailx -s "mail-hello" $USER < tmp-message echo "Message sent."

37. count-files.sh - Script to Count Files #!/bin/bash # # Counts how many files are in the directories passed # as parameter. If not directories are passed it uses # the current directory. # If no arguments use only current directory if [ $# -lt 1 ] then dirs=. else dirs=$* fi #Initialize file counter to 0 count=0

38. Script to count files (cont.) # for all the directories passed as argument for dir in $dirs do echo $dir: for file in $dir/* do echo "$count: $file" count=`expr $count + 1` done echo "$count files found"

39. Review Know what the following commands do ls, cp, mv, rm, mkdir, rmdir, cat, more, less, man, grep, which, whereis Given a shell script, can tell what it does.