BILKENT UNIVERSITY DEPARTMENT OF COMPUTER TECHNOLOGY AND INFORMATION SYSTEMS CTIS156 INFORMATION TECHNOLOGIES II CHAPTER 10: ADVANCED FILE PROCESSING.

Slides:



Advertisements
Similar presentations
UNIX Chapter 10 Advanced File Processing Mr. Mohammad Smirat.
Advertisements

UNIX Chapter 12 Redirection and Piping Mr. Mohammad Smirat.
CIS 118 – Intro to UNIX Shells 1. 2 What is a shell? Bourne shell – Developed by Steve Bourne at AT&T Korn shell – Developed by David Korn at AT&T C-shell.
NETW-240 Shells Last Update Copyright Kenneth M. Chipps Ph.D. 1.
CIS 240 Introduction to UNIX Instructor: Sue Sampson.
Linux+ Guide to Linux Certification, Second Edition
CS 497C – Introduction to UNIX Lecture 21: - The Shell Chin-Chih Chang
Linux+ Guide to Linux Certification, Second Edition
The Unix Shell. Operating System shell The shell is a command interpreter It forms the interface between a user and the operating system When you log.
Guide To UNIX Using Linux Third Edition
CSCI 330 T HE UNIX S YSTEM File operations. OPERATIONS ON REGULAR FILES 2 CSCI The UNIX System Create Edit Display Contents Display Contents Print.
Unix Filters Text processing utilities. Filters Filter commands – Unix commands that serve dual purposes: –standalone –used with other commands and pipes.
Chapter 4: UNIX File Processing Input and Output.
©NIIT Pipes and Filters Lesson 2B / Slide 1 of 28 Introduction to Linux Pre-Assessment Questions 1.Consider the following statements: Statement A: A text.
Chapter 9 Part II Linux Command Line Access to Linux Authenticated login using a Linux account is required to access a Linux system. The Linux prompt will.
3 File Processing Mauro Jaskelioff. Introduction More UNIX commands for handling files Regular Expressions and Searching files Redirection and pipes Bash.
June 1, 1999Manipulating Files1 Introduction to UNIX E. Manipulating Files.
Lesson 11-Locating, Printing, and Archiving User Files.
Week 7 Working with the BASH Shell. Objectives  Redirect the input and output of a command  Identify and manipulate common shell environment variables.
Agenda User Profile File (.profile) –Keyword Shell Variables Linux (Unix) filters –Purpose –Commands: grep, sort, awk cut, tr, wc, spell.
File Processing. Introduction More UNIX commands for handling files Regular Expressions and Searching files Redirection and pipes Bash facilities.
Chapter Four UNIX File Processing. 2 Lesson A Extracting Information from Files.
Agenda Basic Shell Operations Standard Input / Output / Error Redirection of Standard Input / Output / Error ( >, >>,
CIT 140: Introduction to ITSlide #1 CSC 140: Introduction to IT I/O Redirection.
June 1, 1999Using The C-Shell1 Introduction to UNIX F. Using C Shell Features.
Guide To UNIX Using Linux Fourth Edition
BIF703 stdin, stdout, stderr Redirection. stdin, stdout, stderr Recall the Unix philosophy “do one thing well”. Unix has over one thousand commands (utilities)
The “File System” Under UNIX, (almost) everything is a “file”: –Normal files –Directories –Hardware –Sockets –Pipes Things that are not files: –Users –Groups.
– Introduction to the Shell 10/1/2015 Introduction to the Shell – Session Introduction to the Shell – Session 2 · Permissions · Users.
Jozef Goetz, expanded by Jozef Goetz, 2009 Credits: Parts of the slides are based on slides created by UNIX textbook authors, Syed M. Sarwar, Robert.
Title Slide CSS 404/504 The UNIX Operating System (2) By Ralph B. Bisland, Jr.
The UNIX Shell. The Shell Program that constantly runs at terminal after a user has logged in. Prompts the user and waits for user input. Interprets command.
Summer 2015 SILICON VALLEY UNIVERSITY CONFIDENTIAL 1 Introduction to UNIX / Linux - 7 Dr. Jerry Shiao, Silicon Valley University.
Linux Operations and Administration
Jozef Goetz, expanded by Jozef Goetz, 2009 Credits: Parts of the slides are based on slides created by UNIX textbook authors, Syed M. Sarwar, Robert.
The Shell Chapter 7. Overview The Command Line Standard IO Redirection Pipes Running a Program in the Background Killing (a process!)
System Administration Introduction to Unix Session 2 – Fri 02 Nov 2007 Reference:  chapter 1, The Unix Programming Environment, Kernighan & Pike, ISBN.
Summer 2015 SILICON VALLEY UNIVERSITY CONFIDENTIAL 1 Introduction to UNIX / Linux - 6 Dr. Jerry Shiao, Silicon Valley University.
Workbook 5 - Part 2 The Linux Filesystem RH030 Linux Computing Essentials.
Guide to Linux Installation and Administration, 2e1 Chapter 7 The Role of the System Administrator.
Chapter 10 Advanced File Processing. Regular Expressions A compact notation for representing patterns in strings Used by many common Linux utilities such.
Linux+ Guide to Linux Certification, Third Edition
UNIX Commands. Why UNIX Commands Are Noninteractive Command may take input from the output of another command (filters). May be scheduled to run at specific.
Linux+ Guide to Linux Certification Chapter Eight Working with the BASH Shell.
Module 6 – Redirections, Pipes and Power Tools.. STDin 0 STDout 1 STDerr 2 Redirections.
Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.1 © Copyright IBM Corporation 2008 Unit 10 Linux.
Agenda Regular Expressions (Appendix A in Text) –Definition / Purpose –Commands that Use Regular Expressions –Using Regular Expressions –Using the Replacement.
BIF713 Additional Utilities. Linux Utilities  You have learned many Linux commands. Here are some more that you can use:  Data Manipulation (Reg Exps)
Unix/Linux cs3353. The Shell The shell is a program that acts as the interface between the user and the kernel. –The shell is fully programmable and will.
Chapter Four I/O Redirection1 System Programming Shell Operators.
UNIX An Introduction. Brief History UNIX UNIX Created at Bell Labs, 1969 Created at Bell Labs, 1969 BSD during mid 70s BSD during mid 70s AT&T began offering.
Operating Systems Lecture 10. Agenda for Today Review of previous lecture Input, output, and error redirection in UNIX/Linux FIFOs in UNIX/Linux Use of.
Week 9 - Nov 7, Week 9 Agenda I/O redirection I/O redirection pipe pipe tee tee.
Agenda Basic Unix Commands (Chapters 2 & 3) Miscellaneous Commands: which, passwd, date, ps / kill Working with Files: file, touch, cat, more, less, grep,
Linux Commands C151 Multi-User Operating Systems.
Linux+ Guide to Linux Certification, Second Edition Chapter 4 Exploring Linux Filesystems.
Linux+ Guide to Linux Certification, Second Edition
Jozef Goetz, expanded by Jozef Goetz, 2008 Credits: Parts of the slides are based on slides created by UNIX textbook authors, Syed M. Sarwar, Robert.
File Processing. Introduction More UNIX commands for handling files Regular Expressions and Searching files Redirection and pipes Bash facilities.
File Management commands cat Cat command cat cal.txt cat command displays the contents of a file here cal.txt on screen (or standard out).
Linux Administration Working with the BASH Shell.
Linux 201 Training Module Linux Adv File Mgmt.
Lesson 5-Exploring Utilities
Advanced File Processing
C151 Multi-User Operating Systems
stdin, stdout, stderr Redirection
CSE 303 Concepts and Tools for Software Development
Chapter Four UNIX File Processing.
Linux Shell Script Programming
Lecture 4 Redirecting standard I/O & Pipes
Presentation transcript:

BILKENT UNIVERSITY DEPARTMENT OF COMPUTER TECHNOLOGY AND INFORMATION SYSTEMS CTIS156 INFORMATION TECHNOLOGIES II CHAPTER 10: ADVANCED FILE PROCESSING

CH.10 ADVANCED FILE PROCESSING ● Objectives: – To explain how file compression is performed in Linux – To discuss searching for commands and files in Linux file structure – To describe searching files for expressions, strings, and patterns – To explain command history – To describe input/output redirection – To explain Linux pipes 2 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files – Reducing the size of files: ● To save disk space ● To save time when transmitting over the internet ● To save time during file save – Linux provides several commands to compress/decompress files: ● gzip command is used to compress files ● gunzip command is used to decompress files 3 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files (cont) – gzip [options] [file-list] ● It saves the compressed file with a.gz extension ● -f option forces compression even though.gz file already exists ● -r option recursively compresses files in the subdirectories ● -l option displays a list of compressed files, compressed and uncompressed file sizes, compression ratio, etc. 4 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● 10.3 Compressing Files (cont) – gunzip [options] filename ● Uncompresses the compressed files. 5 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING 6 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files – Order set of items according to some criteria. – The sorting process is based on using a field, or portion of each item, known as sort key. – The Linux sort utility can be used to sort items in text (ASCII) files. – By default, the sort utility takes each line, starting with the first column to be the key.

CH.10 ADVANCED FILE PROCESSING 7 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) vural]$ cat students John Johnsen Hassaan Sarwar David Kendal Kelly Kimberly Maham Sarwar

CH.10 ADVANCED FILE PROCESSING 8 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) vural]$ sort students David Kendal Hassaan Sarwar John Johnsen Kelly Kimberly Maham Sarwar

CH.10 ADVANCED FILE PROCESSING 9 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) vural]$ sort +1 students John Johnsen David Kendal Kelly Kimberly Hassaan Sarwar Maham Sarwar

CH.10 ADVANCED FILE PROCESSING 10 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) vural]$ sort +3 students David Kendal Hassaan Sarwar John Johnsen Kelly Kimberly Maham Sarwar

CH.10 ADVANCED FILE PROCESSING 11 CTIS156 INFORMATION TECHNOLOGIES II ● 10.4 Sorting Files (cont) vural]$ sort -r students Maham Sarwar Kelly Kimberly John Johnsen Hassaan Sarwar David Kendal

CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files: – find directory-list expression ● Searches for the files that match the criteria defined by the expression in the directory-list ● Searches the directories recursively ● Expression examples: – -name pattern (for files that match pattern) – -newer file (for files newer than the file) – -user username (for files owned by a specific user) – -print (displays the pathnames of the files found) 12 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files (cont): – Examples of find command: ● find ~ -name vural.c –print ● find / -name profile –print ● find /usr –name socket.h –print ● find. –user ctis -print 13 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● Searching For Commands and Files (cont): – whereis [options] [file-list] ● It searches and displays the location of a command’s files (executable, source and man pages) ● Example: – whereis ls – which [command-list] ● If there are more than one version of a command file, then it displays the absolute pathname of the command executed by the shell. 14 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● Searching a text in the text files: – grep [options] pattern [file-list] ● It searches the pattern (a string) inside the text files in the file-list and if finds, displays the lines that has ● -n (display the line numbers in the output) ● -l (display only the filenames that contain the string) ● Examples: – grep umask /etc/profile – grep include *.c 15 CTIS156 INFORMATION TECHNOLOGIES II

CH.10 ADVANCED FILE PROCESSING ● Command History: – history [options] [filename] ● Displays the command history ● The command history is stored in a file ~/.bash_history ● If a number is specified as an option, the command displays only the last number of commands ● !! (executes the last command in the history list) ● !n (executes the command in line n of the history list) 16 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.1 Introduction: – Most commands and programs perform the following operations: ● Input ● Processing ● Output – Questions: ● Where does a program take it's input? ● Where does it send it's output ● Where are the error messages displayed? 17 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.2 Standard Files: – By default, three standard files are opened automatically by the kernel for every command or program executed: ● stdin (keyboard): Where it takes input ● stdout (screen): Where it sends output ● stderr (screen): Where it sends the error messages 18 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.3 Input Redirection: – The < (less than) symbol is used to change the standard input. The syntax is: – command < input-file – The command takes its input from another file, instead of stdin (keyboard) 19 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● Input Redirection example: - bc is a calculator program in Unix, takes the input form the keyboard by default. - We can write the input to a file and use input redirection $ bc $ cat bc_input 2+2 $ bc < bc_input 4 20 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.4 Output Redirection: – The > (greater than) symbol is used to change the standard output. The syntax is: – command > output-file – The command write its output to an external file instead of the standard output (screen) 21 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● Output Redirection example: $ ls -l > ls.file $ cat ls.file total 30 drwx--x--x 8 vpolat ctisac 1536 Apr 20 10:55. drwxr-xr-x 29 root other 512 Feb 2 12:07.. -rw-r--r-- 1 vpolat ctisac 53 Feb 28 10:57.profile -rw vpolat ctisac 2748 Apr 20 10:55.sh_history drwxr-xr-x 2 vpolat ctisac 512 Apr _fpclass drwxr-xr-x 3 vpolat ctisac 512 Sep _private drwxr-xr-x 3 vpolat ctisac 512 Apr _themes -rw-r--r-- 1 vpolat ctisac 4 Apr 20 10:44 bc_input drwxr-xr-x 2 vpolat ctisac 512 Sep images -rw-r--r-- 1 vpolat ctisac 0 Apr 20 10:55 ls.file drwx vpolat ctisac 512 Jul mail drwxr-xr-x 7 vpolat ctisac 1536 Mar 23 09:30 public_html 22 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.5 Combining Input and Output Redirection: Command output-file Command > output-file < input-file Example: 23 CTIS156 INFORMATION TECHNOLOGIES II $ cat bc_input 2+2 $ bc bc_output $ cat bc_output 4

CH.12 REDIRECTION AND PIPING ● 12.7 Redirecting Standard Error: – The 2> (2 greater than) symbol is used to change the standard error file. The syntax is: – command 2> error-file – The error messages are written into an external file instead of the stderr (screen) 24 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● 12.9 Redirecting stdin stdout and stderr in one command: ● 0< standard input ● 1> standard output ● 2> standard error Example: ● $sort 0 students.sorted 2> sort.error 25 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● Linux Pipes: – In Linux stdout of a command can be connected to the stdin of another command. – | (pipe) character is used between the commands – Syntax: ● command1 | command2 |...| commandN 26 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● Linux Pipes (cont): – Linux pipe example: ● $ls -l | more – The output of ls -l command is fed to more command as input ● $history | tail -7 – The output of history command is fed to tail -7 command as input 27 CTIS156 INFORMATION TECHNOLOGIES II

CH.12 REDIRECTION AND PIPING ● References : ● LINUX: The Textbook. Sarwar, Koretsky, Sarwar. Addison Wesley CTIS156 INFORMATION TECHNOLOGIES II