1. CISC3130: Review of Commands
A review
Spring 2011

2. Review: File systems Absolute pathname, relative pathname
Commands: cd, pwd, mkdir, rmdir, rm, mv
File examination: cat, head, tail, more, less, wc, file
File manipulation: chmod, editors (vi, emacs, pico)
Directory exploration: ls
Understand the long listing format
Different types of file
Regular file:
Directory:
Links: …
Understand permission setting of file/directory
Compare text files files: diff, comp

3. Review: system info & misc
/etc/passwd: stores user accounts in the system
Different fields separated by :
Understand the meaning of the fields
who, who am i: list the users currently logged on
ps: list processes running in the system
date: current date
sleep: resume executing next command after delaying for specified amount of time

4. Shell related commands
chsh: change default shell to use
Startup configuration files for bash
.bash_profile,
.bash_rc
To find out the name of the shell that you are using:
From command line, type
echo $0
Shell built-in commands
echo
set
alias
read

5. Filters commands For each command: Common behavior:
Understand the basic functionalities of the following commands
Know how to read online manual for more info.
Common behavior:
If no files are specified in command line, process standard input
Output is generally written to standard output
Benefits of such design?

6. filters
sort
cut, paste
comp, diff
uniq tr
grep family

7. grep grep - uses regular expressions for pattern matching
fgrep - file grep, does not use regular expressions, only matches fixed strings but can get search strings from a file
egrep - extended grep, uses a more powerful set of regular expressions but does not support backreferencing, generally fastest

8. Hints on bash scripting
Start with requirement, for example:
“Write a script that reports the number of processes run by each of the users that are currently online”.
Identify sub-tasks:
Find out all users that are currently online
For a given user, find out the number of processes belonging to him/her
Compose solution using solution to sub-task
for user in ## all users currently online do
## find out the number of processes run by $user
done

9. Solve sub-task #1: users online
Search for candidates …
Is there a command that fits this job description exactly?
Well, command who almost fits…
Does the output of the command directly usable ? No
Try the command out …
How to process the output of the command so that it can be used ?
How to select only the first field of output ?
Test and confirm your solution:
who | cut –d ‘ ‘ –f 1

10. Solve subtask #2: number of processes
Task #2: to find out # of processed by a user $user
No command fits this requirement directly…
But command ps can give us all processes in the system …
The output of ps also tells us the user each process belongs to
Command wc can be used to count # of processes, as ps displays one process per line
Here, we are collecting info (using ps), and processing the info in multiple steps using multiple commands
Now you try to put them together …

11. Final solution

12. Another example: Requirement:
“ For all regular files under a directory given in command line, displays what kind of file it is (using command file)”.
Sub-tasks:
Find all regular files under current directory
Psedocode:
dir=$1
for filename in ##list of all regular file under $dir do
file $filename
done

13. Alternatively Psedocode: dir=$1
for filename in ##the list of all files under $dir do
## if the file is regular file
file $filename
done

14. Solving subtask Find all files under a directory dir ls $dir
So we have
dir=$1
for filename in ls $dir do
## if the file is regular file
file $filename
done
Any problem ?

15. Sed: Stream-oriented, Non-Interactive, Text Editor
Look for patterns one line at a time, like grep
Change lines of the file
Non-interactive text editor
Editing commands come in as script
There is an interactive editor ed which accepts the same commands
A Unix filter
Superset of previously mentioned tools

16. Two power filters
sed and awk

17. Introduction to sed: substitution
Substitute “command”: s
changes all occurrences of a regular expression into a new value.
to change "day" in file old to "night" in the "new" file:
sed s/day/night/ <old >new
Generally: replace occurrence of a pattern in standard input, with a given string, and display result in standard output
sed s/regular expression/replace string/

18. Introduction to sed: substitution
If you have meta-characters in the command, quotes are necessary
sed 's/3.1415[0-9]*/PI/' <old >new
To mark a matching pattern
grep –n count mylab1.cpp | sed s/count/<count>/

19. Delimiter
One can use any letter to delimit different parts of command s
If delimiter appears in regular expr or replace str, escape them
To change /usr/local/bin to /common/bin:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
It is easier to read if you use other letter as a delimiter:
sed 's_/usr/local/bin_/common/bin_' <old >new
sed 's:/usr/local/bin:/common/bin:' <old >new
sed 's|/usr/local/bin|/common/bin|' <old >new

20. How sed works? sed, like most Unix utilties, read a line at a time
By default, sed command applies to the first occurrence of the pattern in a line.
~]$ sed 's/aa*/bb/'
ab ab
bbb ab
To apply to every occurrence, use option g (global)
sed 's/aa*/bb/g‚

21. How sed works: aggressive matching
sed finds the longest string in the line that matches the pattern, and substitute it with the replacing string
Pattern aa* matches with 1 or more a’s
~]$ sed 's/aa*/bb/'
aaab
bbb

22. Substitution with referencing
How to mark all numbers (integers or floating points) using angled brackets?
E.g., 28 replaced by <28>, replaced by <3.1415>
Use special character "&“, which refer to string that matches the pattern (similar to backreference in grep.)
sed 's/[0-9][0-9]*\.[0-9]*/(&)/g'
You can have any number of "&" in replacement string.
You could also double a pattern, e.g. the first number of a line:
$echo "123 abc" | sed 's/[0-9]*/& &/'
abc

23. Multiple commands
To combine multiple commands, use -e before each command:
sed -e 's/a/A/' -e 's/b/B/' <old >new
If you have a large number of sed commands, you can put them into a file, say named as sedscript
# sed comment - This script changes lower case vowels to upper case
s/a/A/g
s/e/E/g
s/i/I/g
s/o/O/g
s/u/U/g
each command must be on a separate line.
Invoke sed with a script:
sed -f sedscript <file.txt >file_cap.txt

24. sed interpreter script
Alternatively, starts the script file (named CapVowel) with
#!/bin/sed -f s/a/A/g s/e/E/g s/i/I/g s/o/O/g s/u/U/g and make the file executable
Then you can evoke it directly:
CapVowel <old >new

25. Restrict operations Restrict commands to certain lines
Specifying a line by its number.
sed '3 s/[0-9][0-9]*//' <file >new
Specifying a range of lines by number.
sed '1,100 s/A/a/' All lines containing a pattern.
To delete first number on all lines that start with a "#," use:
sed '/^#/ s/[0-9][0-9]*//'
Many other ways to restrict

26. Command d
A useful command deletes every line that matches the restriction: "d.“
To look at the first 10 lines of a file, you can use:
sed '11,$ d' <file
i.e., delete from line 11 to the end of the file
If you want to chop off the header of a mail message, which is everything up to the first blank line, use:
sed '1,/^$/ d' <file

27. Command q Ex: another way to duplicate the head command is:
This command is most useful when you wish to abort the editing after some condition is reached.
Ex: another way to duplicate the head command is:
sed '11 q' which quits when the eleventh line is reached.

28. Backreference
To keep first word of a line, and delete the rest of the line, mark first word with the parenthesis:
sed 's/\([a-z]*\).*/\1/'
Recall: regular exprssions are greedy, and try to match as much as possible.
"[a-z]*" matches zero or more lower case letters, and tries to be as big as possible.
".*" matches zero or more characters after the first match. Since the first one grabs all of the lower case letters, the second matches anything else.
Ex:
$echo abcd123 | sed 's/\([a-z]*\).*/\1/'
abcd

29. Backreference (cont’d)
If you want to switch two words around, you can remember two patterns and change the order around:
sed 's/\([a-z][a-z]*\) \([a-z][a-z]*\)/\2 \1/’
To eliminate duplicated words:
sed 's/\([a-z]*\) \1/\1/'
If you want to detect duplicated words, you can use
sed -n '/\([a-z][a-z]*\) \1/p’
Numeric value can have up to nine values: "\1" thru "\9.“
To reverse first three characters on a line, you can use
sed 's/^\(.\)\(.\)\(.\)/\3\2\1/'

30. Sed commands & scripts
Each sed command consists of up to two addresses and an action, where the address can be a regular expression or line number.
A script is nothing more than a file of commands
address
action
command
address
action
address
action
address
action
address
action
script

31. sed: a conceptual overview
All editing commands in a sed script are applied in order to each input line.
If a command changes input, subsequent command address will be applied to current (modified) line in the pattern space, not original input line.
Original input file is unchanged (sed is a filter), and the results are sent to standard output (but can be redirected to a file).

32. Why awk? It is an excellent filter and report writer.
Many UNIX utilities generates rows and columns of information.
AWK is an excellent tool for processing these rows and columns, and is easier to use AWK than most conventional programming languages.
A. Aho, B. W. Kernighan and P. Weinberger.

33. Features of awk a pseudo-C interpretor,
it understands same arithmetic operators as C
string manipulation functions
search for particular strings and modify output.
AWK also has associative arrays, which are incredible useful, and is a feature most computing languages lack. Associative arrays can make a complex problem a trivial exercise.