A Guide to Unix Using Linux Fourth Edition

Slides:



Advertisements
Similar presentations
A Guide to Unix Using Linux Fourth Edition
Advertisements

A Guide to Unix Using Linux Fourth Edition
Linux+ Guide to Linux Certification, Second Edition
Linux+ Guide to Linux Certification, Second Edition
Guide To UNIX Using Linux Third Edition
T UTORIAL OF U NIX C OMMAND & SHELL SCRIPT S 5027 Professor: Dr. Shu-Ching Chen TA: Samira Pouyanfar Spring 2015.
Guide To UNIX Using Linux Third Edition
Guide To UNIX Using Linux Third Edition
Guide To UNIX Using Linux Third Edition
Guide to Linux Installation and Administration, 2e1 Chapter 6 Using the Shell and Text Files.
Introduction to Unix (CA263) Introduction to Shell Script Programming By Tariq Ibn Aziz.
Lecture 02CS311 – Operating Systems 1 1 CS311 – Lecture 02 Outline UNIX/Linux features – Redirection – pipes – Terminating a command – Running program.
Grep, comm, and uniq. The grep Command The grep command allows a user to search for specific text inside a file. The grep command will find all occurrences.
Introduction to UNIX GPS Processing and Analysis with GAMIT/GLOBK/TRACK T. Herring, R. King. M. Floyd – MIT UNAVCO, Boulder - July 8-12, 2013 Directory.
CSCI 330 T HE UNIX S YSTEM File operations. OPERATIONS ON REGULAR FILES 2 CSCI The UNIX System Create Edit Display Contents Display Contents Print.
Unix Filters Text processing utilities. Filters Filter commands – Unix commands that serve dual purposes: –standalone –used with other commands and pipes.
UNIX Filters.
Shell Script Examples.
Linux Commands LINUX COMMANDS.
Chapter 4: UNIX File Processing Input and Output.
Chapter Seven Advanced Shell Programming. 2 Lesson A Developing a Fully Featured Program.
Advanced File Processing
Linux environment ● Graphical interface – X-window + window manager ● Text interface – terminal + shell.
Introduction to Shell Script Programming
Week 7 Working with the BASH Shell. Objectives  Redirect the input and output of a command  Identify and manipulate common shell environment variables.
Agenda User Profile File (.profile) –Keyword Shell Variables Linux (Unix) filters –Purpose –Commands: grep, sort, awk cut, tr, wc, spell.
Chapter Four UNIX File Processing. 2 Lesson A Extracting Information from Files.
Guide To UNIX Using Linux Fourth Edition
LIN 6932 Unix Lecture 6 Hana Filip. LIN 6932 HW6 - Part II solutions posted on my website see syllabus.
A Guide to Unix Using Linux Fourth Edition
Introduction to Unix (CA263) File Processing. Guide to UNIX Using Linux, Third Edition 2 Objectives Explain UNIX and Linux file processing Use basic file.
Unix programming Term: III B.Tech II semester Unit-II PPT Slides Text Books: (1)unix the ultimate guide by Sumitabha Das (2)Advanced programming.
Guide to Linux Installation and Administration, 2e1 Chapter 7 The Role of the System Administrator.
Advanced File Processing. 2 Objectives Use the pipe operator to redirect the output of one command to another command Use the grep command to search for.
Linux+ Guide to Linux Certification, Third Edition
UNIX Shell Script (1) Dr. Tran, Van Hoai Faculty of Computer Science and Engineering HCMC Uni. of Technology
Chapter Five Advanced File Processing Guide To UNIX Using Linux Fourth Edition Chapter 5 Unix (34 slides)1 CTEC 110.
Chapter Five Advanced File Processing. 2 Objectives Use the pipe operator to redirect the output of one command to another command Use the grep command.
Module 6 – Redirections, Pipes and Power Tools.. STDin 0 STDout 1 STDerr 2 Redirections.
Agenda Regular Expressions (Appendix A in Text) –Definition / Purpose –Commands that Use Regular Expressions –Using Regular Expressions –Using the Replacement.
Introduction to Unix – CS 21 Lecture 12. Lecture Overview A few more bash programming tricks The here document Trapping signals in bash cut and tr sed.
Review Chapters 5 thru 8. 2 Two Groups of Commands Select commands Manipulate and Format commands.
Introduction to Unix (CA263) File Processing (continued) By Tariq Ibn Aziz.
Chapter Five Advanced File Processing. 2 Lesson A Selecting, Manipulating, and Formatting Information.
Searching and Sorting. Why Use Data Files? There are many cases where the input to the program may come from a data file.Using data files in your programs.
Chapter Six Introduction to Shell Script Programming.
Lesson 3-Touring Utilities and System Features. Overview Employing fundamental utilities. Linux terminal sessions. Managing input and output. Using special.
– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session Introduction to the Shell – Session 3 · Job control · Start,
Linux+ Guide to Linux Certification, Second Edition Chapter 4 Exploring Linux Filesystems.
Linux+ Guide to Linux Certification, Second Edition
ORAFACT Text Processing. ORAFACT Searching Inside Files grep - searches for patterns within files grep [options] [[-e] pattern] filename [...] -n shows.
Uniq The uniq command is useful when you need to find duplicate lines in a file. The basic format of the command is uniq in_file out_file In this format,
Lesson 6-Using Utilities to Accomplish Complex Tasks.
CS 403: Programming Languages Lecture 20 Fall 2003 Department of Computer Science University of Alabama Joel Jones.
Linux Administration Working with the BASH Shell.
Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the.
SIMPLE FILTERS. CONTENTS Filters – definition To format text – pr Pick lines from the beginning – head Pick lines from the end – tail Extract characters.
CSE 303 Concepts and Tools for Software Development Richard C. Davis UW CSE – 10/9/2006 Lecture 6 – String Processing.
Tutorial of Unix Command & shell scriptS 5027
Lesson 5-Exploring Utilities
Chapter 6 Filters.
Guide To UNIX Using Linux Third Edition
INTRODUCTION TO UNIX: The Shell Command Interface
Tutorial of Unix Command & shell scriptS 5027
Tutorial of Unix Command & shell scriptS 5027
Guide To UNIX Using Linux Third Edition
Tutorial of Unix Command & shell scriptS 5027
Chapter Four UNIX File Processing.
More advanced BASH usage
Linux Commands LINUX COMMANDS.
Presentation transcript:

A Guide to Unix Using Linux Fourth Edition Chapter 5 Advanced File Processing

Objectives Use the pipe operator to redirect the output of one command to another command Use the grep command to search for a specified pattern in a file Use the uniq command to remove duplicate lines from a file Use the comm and diff commands to compare two files A Guide to Unix Using Linux, Fourth Edition

Objectives (continued) Use the wc command to count words, characters, and lines in a file Use manipulation and transformation commands, which include sed, tr, and pr Design a new file-processing application by creating, testing, and running shell scripts A Guide to Unix Using Linux, Fourth Edition

Advancing Your File-Processing Techniques Commands used for file processing can be organized into two categories: Selection commands Focus on extracting specific information from files Manipulation and transformation commands Alter and transform extracted information into useful and appealing formats A Guide to Unix Using Linux, Fourth Edition

Advancing Your File-Processing Techniques (continued) A Guide to Unix Using Linux, Fourth Edition

Advancing Your File Processing Techniques (continued) A Guide to Unix Using Linux, Fourth Edition

Using the Selection Commands The pipe (|) operator: another redirection operator Some useful selection commands: grep diff uniq comm wc A Guide to Unix Using Linux, Fourth Edition

Using the Pipe Operator < and > can be used for redirection Pipe operator (|) redirects the output of one command to the input of another command first_command | second_command Pipe can connect several commands first_command | second_command | third_command ... A Guide to Unix Using Linux, Fourth Edition

Using the Pipe Operator (continued) A Guide to Unix Using Linux, Fourth Edition

Using the grep Command Example: May be enclosed in single/double quotes Example: grep -r Computer Resources Committee /documentation A Guide to Unix Using Linux, Fourth Edition

Using the uniq Command uniq removes duplicate lines from a file Compares only consecutive lines Requires sorted input A Guide to Unix Using Linux, Fourth Edition

Using the uniq Command (continued) Consider a simple file called parts that contains the following entries: muffler shocks alternator battery radiator coil spark plugs A Guide to Unix Using Linux, Fourth Edition

Using the uniq Command (continued) A Guide to Unix Using Linux, Fourth Edition

Using the uniq Command (continued) A Guide to Unix Using Linux, Fourth Edition

Using the comm Command Like uniq, comm identifies duplicate lines Unlike uniq: Does not delete duplicates Works with two files rather than one A Guide to Unix Using Linux, Fourth Edition

Using the diff Command Commonly used to determine the minimal set of changes needed to convert file1 to file2 Differing text preceded by < or > A Guide to Unix Using Linux, Fourth Edition

Using the diff Command (continued) Monkeys:Bananas:2000:850.00 Lions:Raw Meat:4000:1245.50 Camels:Vegetables:2300:564.75 Elephants:Hay:120000:1105.75 File zoo1 contains: File zoo2 contains: diff zoo1 zoo2 diff zoo2 zoo1 Monkeys:Bananas:2000:850.00 Lions:Raw Meat:4000:1245.50 Camels:Vegetables:2300:564.75 Elephants:Hay:120000:1105.75 3d2 < Lions:Raw Meat:4000:1245.50 … 2a3 > Lions:Raw Meat:4000:1245.50 … A Guide to Unix Using Linux, Fourth Edition

Using the wc Command You can specify all three options in the command line (e.g., -lwc) If entered without options, you see counts of lines, words, and characters in that order A Guide to Unix Using Linux, Fourth Edition

Using the wc Command (continued) A Guide to Unix Using Linux, Fourth Edition

Using Manipulation and Transformation Commands Several commands can be used to edit and transform data’s appearance: sed tr pr A Guide to Unix Using Linux, Fourth Edition

Introducing the sed Command A Guide to Unix Using Linux, Fourth Edition

Translating Characters Using the tr Command A popular use is to convert lowercase characters to uppercase characters Some examples: tr "c" " " < constants tr ’c’ ’ ’ < constants A Guide to Unix Using Linux, Fourth Edition

Using the pr Command to Format Your Output If no file is specified or “-” is specified as the file, reads the standard input Default output: single-column pages of 66 lines Each page has a five-line header A Guide to Unix Using Linux, Fourth Edition

Designing a New File Processing Application Files, records, and fields are logical structures How you set up records in a file can influence what you can do with an application Also affects the ways in which you can use selection/manipulation/transformation commands Selection of fields is important for enabling useful sorts and for linking of files (join) An ID can be a key field for sorting A Guide to Unix Using Linux, Fourth Edition

Designing Records First task in record design phase: define fields Record layout identifies each field by name and data type Design file record to store only those fields relevant to each record’s primary purpose Short records are preferable Must include a field that uniquely identifies each record in the file A Guide to Unix Using Linux, Fourth Edition

Linking Files with Keys Multiple files can be joined by a key Key: common field shared by each of the linked files Plan a way to join files in design phase A Guide to Unix Using Linux, Fourth Edition

Linking Files with Keys (continued) A Guide to Unix Using Linux, Fourth Edition

Creating the Programmer and Project Files A Guide to Unix Using Linux, Fourth Edition

Creating the Programmer and Project Files (continued) A Guide to Unix Using Linux, Fourth Edition

Formatting Output printf function within awk is used to format output A Guide to Unix Using Linux, Fourth Edition

Formatting Output (continued) A Guide to Unix Using Linux, Fourth Edition

Using a Shell Script to Implement the Application Tip: test and debug each command before you place it in your script file Use vi or Emacs to create script files Commenting shell scripts is crucial Helps creator and other programmers Use pound (#) character A Guide to Unix Using Linux, Fourth Edition

Running a Shell Script You can run a shell script in virtually any shell We will use Bash Two easy ways to run scripts: Call the interpreter: sh testscript Can accompany it with several debugging options Type ./ in front of name: ./testscript Must make script executable first Use chmod to add x permission Advice: specify with what shell your script is intended to be used Example: #!/bin/bash A Guide to Unix Using Linux, Fourth Edition

Putting It All Together to Produce the Report Combine small scripts into a larger script file Convenient Complete a large task by dividing it into a series of smaller ones Test each small script independently A Guide to Unix Using Linux, Fourth Edition

Summary Selection commands extract information Manipulation and transformation commands alter extracted information into useful/appealing formats grep searches for a specific pattern in a file uniq removes duplicate lines from a file comm compares lines common to two different files and produces three-column output with the variances diff attempts to determine the minimum set of changes needed to convert the contents of one file to match the contents of another file A Guide to Unix Using Linux, Fourth Edition

Summary (continued) wc counts bytes, words, or lines in a file sed is a stream editor designed to make global changes to large files tr copies data read from the standard input to the standard output, substituting or deleting the characters specified by options and patterns pr prints the standard output in pages When designing a file-processing application, define logical structures (including record layout) Shell scripts should be commented and simple A Guide to Unix Using Linux, Fourth Edition

Command Summary A Guide to Unix Using Linux, Fourth Edition

Command Summary (continued) A Guide to Unix Using Linux, Fourth Edition