2. BILKENT UNIVERSITY DEPARTMENT OF COMPUTER TECHNOLOGY AND INFORMATION SYSTEMS CTIS156 INFORMATION TECHNOLOGIES II CHAPTER 10: ADVANCED FILE PROCESSING

3. CH.10 ADVANCED FILE PROCESSING ● Objectives: – To explain how file compression is performed in Linux – To discuss searching for commands and files in Linux file structure – To describe searching files for expressions, strings, and patterns – To explain command history – To describe input/output redirection – To explain Linux pipes 2 CTIS156 INFORMATION TECHNOLOGIES II

4. CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files – Reducing the size of files: ● To save disk space ● To save time when transmitting over the internet ● To save time during file save – Linux provides several commands to compress/decompress files: ● gzip command is used to compress files ● gunzip command is used to decompress files 3 CTIS156 INFORMATION TECHNOLOGIES II

5. CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files (cont) – gzip [options] [file-list] ● It saves the compressed file with a.gz extension ● -f option forces compression even though.gz file already exists ● -r option recursively compresses files in the subdirectories ● -l option displays a list of compressed files, compressed and uncompressed file sizes, compression ratio, etc. 4 CTIS156 INFORMATION TECHNOLOGIES II

6. CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files (cont) – gunzip [options] filename ● Uncompresses the compressed files. 5 CTIS156 INFORMATION TECHNOLOGIES II

7. CH.10 ADVANCED FILE PROCESSING 6 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files – Order set of items according to some criteria. – The sorting process is based on using a field, or portion of each item, known as sort key. – The Linux sort utility can be used to sort items in text (ASCII) files. – By default, the sort utility takes each line, starting with the first column to be the key.

8. CH.10 ADVANCED FILE PROCESSING 7 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) [vural@vpdesktop vural]$ cat students John Johnsen john.johnsen@tp.com 503.555.1111 Hassaan Sarwar hsarwar@k12.st.or 503.444.2132 David Kendal d_kendal@msnbc.org 229.111.2013 Kelly Kimberly kellyk@umich.gov 555.123.9999 Maham Sarwar msrwar@k12.st.or 713.888.0000

9. CH.10 ADVANCED FILE PROCESSING 8 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) [vural@vpdesktop vural]$ sort students David Kendal d_kendal@msnbc.org 229.111.2013 Hassaan Sarwar hsarwar@k12.st.or 503.444.2132 John Johnsen john.johnsen@tp.com 503.555.1111 Kelly Kimberly kellyk@umich.gov 555.123.9999 Maham Sarwar msrwar@k12.st.or 713.888.0000

10. CH.10 ADVANCED FILE PROCESSING 9 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) [vural@vpdesktop vural]$ sort +1 students John Johnsen john.johnsen@tp.com 503.555.1111 David Kendal d_kendal@msnbc.org 229.111.2013 Kelly Kimberly kellyk@umich.gov 555.123.9999 Hassaan Sarwar hsarwar@k12.st.or 503.444.2132 Maham Sarwar msrwar@k12.st.or 713.888.0000

11. CH.10 ADVANCED FILE PROCESSING 10 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) [vural@vpdesktop vural]$ sort +3 students David Kendal d_kendal@msnbc.org 229.111.2013 Hassaan Sarwar hsarwar@k12.st.or 503.444.2132 John Johnsen john.johnsen@tp.com 503.555.1111 Kelly Kimberly kellyk@umich.gov 555.123.9999 Maham Sarwar msrwar@k12.st.or 713.888.0000

12. CH.10 ADVANCED FILE PROCESSING 11 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) [vural@vpdesktop vural]$ sort -r students Maham Sarwar msrwar@k12.st.or 713.888.0000 Kelly Kimberly kellyk@umich.gov 555.123.9999 John Johnsen john.johnsen@tp.com 503.555.1111 Hassaan Sarwar hsarwar@k12.st.or 503.444.2132 David Kendal d_kendal@msnbc.org 229.111.2013

13. CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files: – find directory-list expression ● Searches for the files that match the criteria defined by the expression in the directory-list ● Searches the directories recursively ● Expression examples: – -name pattern (for files that match pattern) – -newer file (for files newer than the file) – -user username (for files owned by a specific user) – -print (displays the pathnames of the files found) 12 CTIS156 INFORMATION TECHNOLOGIES II

14. CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files (cont): – Examples of find command: ● find ~ -name vural.c –print ● find / -name profile –print ● find /usr –name socket.h –print ● find. –user ctis -print 13 CTIS156 INFORMATION TECHNOLOGIES II

15. CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files (cont): – whereis [options] [file-list] ● It searches and displays the location of a command’s files (executable, source and man pages) ● Example: – whereis ls – which [command-list] ● If there are more than one version of a command file, then it displays the absolute pathname of the command executed by the shell. 14 CTIS156 INFORMATION TECHNOLOGIES II

16. CH.10 ADVANCED FILE PROCESSING ● Searching a text in the text files: – grep [options] pattern [file-list] ● It searches the pattern (a string) inside the text files in the file-list and if finds, displays the lines that has ● -n (display the line numbers in the output) ● -l (display only the filenames that contain the string) ● Examples: – grep umask /etc/profile – grep include *.c 15 CTIS156 INFORMATION TECHNOLOGIES II

17. CH.10 ADVANCED FILE PROCESSING ● Command History: – history [options] [filename] ● Displays the command history ● The command history is stored in a file ~/.bash_history ● If a number is specified as an option, the command displays only the last number of commands ● !! (executes the last command in the history list) ● !n (executes the command in line n of the history list) 16 CTIS156 INFORMATION TECHNOLOGIES II

18. CH.12 REDIRECTION AND PIPING ● 12.1 Introduction: – Most commands and programs perform the following operations: ● Input ● Processing ● Output – Questions: ● Where does a program take it's input? ● Where does it send it's output ● Where are the error messages displayed? 17 CTIS156 INFORMATION TECHNOLOGIES II

19. CH.12 REDIRECTION AND PIPING ● 12.2 Standard Files: – By default, three standard files are opened automatically by the kernel for every command or program executed: ● stdin (keyboard): Where it takes input ● stdout (screen): Where it sends output ● stderr (screen): Where it sends the error messages 18 CTIS156 INFORMATION TECHNOLOGIES II

20. CH.12 REDIRECTION AND PIPING ● 12.3 Input Redirection: – The < (less than) symbol is used to change the standard input. The syntax is: – command < input-file – The command takes its input from another file, instead of stdin (keyboard) 19 CTIS156 INFORMATION TECHNOLOGIES II

21. CH.12 REDIRECTION AND PIPING ● Input Redirection example: - bc is a calculator program in Unix, takes the input form the keyboard by default. - We can write the input to a file and use input redirection $ bc 2+2 4 $ cat bc_input 2+2 $ bc < bc_input 4 20 CTIS156 INFORMATION TECHNOLOGIES II

22. CH.12 REDIRECTION AND PIPING ● 12.4 Output Redirection: – The > (greater than) symbol is used to change the standard output. The syntax is: – command > output-file – The command write its output to an external file instead of the standard output (screen) 21 CTIS156 INFORMATION TECHNOLOGIES II

23. CH.12 REDIRECTION AND PIPING ● Output Redirection example: $ ls -l > ls.file $ cat ls.file total 30 drwx--x--x 8 vpolat ctisac 1536 Apr 20 10:55. drwxr-xr-x 29 root other 512 Feb 2 12:07.. -rw-r--r-- 1 vpolat ctisac 53 Feb 28 10:57.profile -rw------- 1 vpolat ctisac 2748 Apr 20 10:55.sh_history drwxr-xr-x 2 vpolat ctisac 512 Apr 16 2004 _fpclass drwxr-xr-x 3 vpolat ctisac 512 Sep 23 2004 _private drwxr-xr-x 3 vpolat ctisac 512 Apr 16 2004 _themes -rw-r--r-- 1 vpolat ctisac 4 Apr 20 10:44 bc_input drwxr-xr-x 2 vpolat ctisac 512 Sep 12 2005 images -rw-r--r-- 1 vpolat ctisac 0 Apr 20 10:55 ls.file drwx------ 2 vpolat ctisac 512 Jul 20 2004 mail drwxr-xr-x 7 vpolat ctisac 1536 Mar 23 09:30 public_html 22 CTIS156 INFORMATION TECHNOLOGIES II

24. CH.12 REDIRECTION AND PIPING ● 12.5 Combining Input and Output Redirection: Command output-file Command > output-file < input-file Example: 23 CTIS156 INFORMATION TECHNOLOGIES II $ cat bc_input 2+2 $ bc bc_output $ cat bc_output 4

25. CH.12 REDIRECTION AND PIPING ● 12.7 Redirecting Standard Error: – The 2> (2 greater than) symbol is used to change the standard error file. The syntax is: – command 2> error-file – The error messages are written into an external file instead of the stderr (screen) 24 CTIS156 INFORMATION TECHNOLOGIES II

26. CH.12 REDIRECTION AND PIPING ● 12.9 Redirecting stdin stdout and stderr in one command: ● 0< standard input ● 1> standard output ● 2> standard error Example: ● $sort 0 students.sorted 2> sort.error 25 CTIS156 INFORMATION TECHNOLOGIES II

27. CH.12 REDIRECTION AND PIPING ● 12.11 Linux Pipes: – In Linux stdout of a command can be connected to the stdin of another command. – | (pipe) character is used between the commands – Syntax: ● command1 | command2 |...| commandN 26 CTIS156 INFORMATION TECHNOLOGIES II

28. CH.12 REDIRECTION AND PIPING ● Linux Pipes (cont): – Linux pipe example: ● $ls -l | more – The output of ls -l command is fed to more command as input ● $history | tail -7 – The output of history command is fed to tail -7 command as input 27 CTIS156 INFORMATION TECHNOLOGIES II

29. CH.12 REDIRECTION AND PIPING ● References : ● LINUX: The Textbook. Sarwar, Koretsky, Sarwar. Addison Wesley. 2002. 28 CTIS156 INFORMATION TECHNOLOGIES II