1. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Module 8 Pipes, Redirection and REGEX

2. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Exam Objective 3.2 Searching and Extracting Data from Files Objective Summary – Piping and redirection – Partial POSIX

3. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Command Line and Redirection

4. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Command Line Pipes The pipe character ( | ) can be used between two commands to send the output of the first as input to the second: – ls /etc | head The output of ls /etc is sent to head as input.

5. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Command Line Pipelines Multiple commands can be combined to form pipelines. The order in which commands are added to the pipeline can affect the output:

6. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. I/O Redirection Three Input/Output (I/O) streams associated with every command: – Standard Input (STDIN) is normally provided by the user via the keyboard. – Standard Output (STDOUT) is the output produced by the command when operating correctly. STDOUT normally appears in the same window as where command executed. – Standard Error (STERR) is the is the output produced by the command when an error has occurred. STDOUT normally appears in the same window as where command executed.

7. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. I/O Redirection Symbols Summary of redirection possible with the bash shell: – < /path/to/file(Redirect STDIN from file) – > /path/to/file(Redirect STDOUT overwriting file) – >> /path/to/file(Redirect STDOUT appending file) – 2> /path/to/file(Redirect STDERR overwriting file) – 2>> /path/to/file(Redirect STDERR appending file) – &> /path/to/file(Redirect STDERR and STDOUT overwriting file) – &>> /path/to/file(Redirect STDERR and STDOUT appending file)

8. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. The null device The null device is represented by the /dev/null file. (Otherwise known as the “Bit Bucket”) This file is very useful in redirection of input and output. This file serves two purposes: – any output redirected to /dev/null is discarded. – /dev/null can be used for input to provide a stream of null values.

9. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDIN, STDOUT, and STDERR

10. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDIN or 0 Standard Input (STDIN) normally is provided by the keyboard but can be redirected with the < symbol. STDIN can be read by programs to get data for them to process. To signal a program that you wish to stop providing data by the keyboard via STDIN, type CTRL-D. The tr command reads its data from STDIN. It translates from one set of characters to another. If you were the user typing the data to be translated by the tr command, you would type CTRL-D when finished.

11. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDIN from keyboard In the following example, the tr command translates from lowercase to uppercase after the user typed the command and pressed Enter. Then, "alpha" was typed and Enter pressed. Finally, the user typed CTRL-D.

12. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Redirecting STDIN from file The tr command translates from lowercase to uppercase with STDIN being redirected from the /etc/hosts file:

13. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDOUT or 1 Standard Out (STDOUT) is the output from the command when operating correctly. It is usually displayed in the same window where the command is executed. The echo command is used to print messages to STDOUT. It can be used to demonstrate how STDOUT can be redirected, as shown on the following slide.

14. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Redirecting STDOUT In the example below, the echo Linux 1 command is executed and the output appears on STDOUT. Then, the echo Linux 1 > a.txt command redirects the output to the file a.txt. Finally, the command cat a.txt sends the file contents to STDOUT, so the output is shown.

15. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Appending STDOUT redirection Using a single arrow > for STDOUT redirection will clobber, or overwrite, the file specified. Using the double arrow >> for STDOUT redirection will either create a new file or append an existing one:

16. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDERR or 2 Standard Error (STDERR) is the output of a command after an error has occurred. It is normally sent to the console/terminal where the command is executed. ls /fake is a command that will cause an error to be output to STDERR because the /fake file does not exist.

17. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Redirecting STDERR ls /fake 2> /tmp/err.msg is a command that would cause an error to be sent to STDERR which is then redirected to the /tmp/err.msg file. The cat /tmp/err.msg command sends the contents of the file to STDOUT to display the file:

18. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Disposing of STDERR ls /fake 2> /dev/null is a command that would cause STDERR to be redirected to the /dev/null file, in effect disposing of the error message. Notice cat /dev/null displays no visible output.

19. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Working with STDERR and STDOUT find is a command that searches the filesystem. It sends output to STDOUT when it successfully locates a file that matches your criteria. It sends output to STDERR when it fails to access a directory. The find command will be used to demonstrate redirecting both STDOUT and STDERR on the following slides. More detail about the find command appears later in this chapter.

20. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. STDERR and STDOUT Example The following example demonstrates the find command searching recursively the /etc/pki directory for any files matching "*.crt". Two lines of STDERR and two lines of STDOUT messages appear:

21. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Isolating STDERR In the next example, the STDOUT output is redirected to the /dev/null file, so the STDERR output alone is sent to the terminal window:

22. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Isolating STDOUT In the next example, the STDERR output is now redirected to /dev/null file, so the STDOUT output alone is sent to the terminal window:

23. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Redirecting Multiple Streams Separately In the next example, the STDERR output is sent to the crt.err file and the STDOUT output is sent to the crt.txt file:

24. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Redirecting Multiple Streams Combined In this example, both STDOUT and STERR are redirected into the same file, crt.all:

25. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. find Command

26. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Searching with find command The filesystem has hundreds of directories with thousands of files making finding files challenging. The find command is a powerful tool to be able to search for files in different ways including: – name – size – date – ownership

27. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Syntax of find command The find command has the following syntax: – find [start_dir] [search_op] [criteria] [result] If the starting directory (start_dir) is not specified, then the current directory is assumed. The search option (search_op) is how the search will be done. For example, use the -name option to search by name.

28. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Syntax of find command (cont'd) The search criteria (criteria) is the data to be used with the search option. So, if the search option was - name, then the search criteria would be the name of the file to find. The result option (result) defaults to -print, which will output the names of the files that are found. Other result options can perform actions on the files that are found.

29. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Searching by file name Consider the following command: find /etc/pki -name "*.crt" Begins searching recursively from the /etc/pki directory Output any file names that match "*.crt" (anything that ends in ".crt“).

30. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Displaying file detail The option -ls will create output similar to the ls -l command. (show both) The columns output are: inode, blocks used, permissions, link count, user owner, group owner, size, date/time, and file name.

31. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Searching by file size The -size option can be used by find to search by its size. Large units can be specified as K, M, G, etc. Using +1M means more than one megabyte. Using -1M means less than one megabyte.

32. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Useful options for find command OptionExampleMeaning -maxdepth-maxdepth 1 Only search specified directory and its immediate subdirectories -group-group payroll Find any files owned by the payroll group -iname-iname hosts Case insensitive search by filename -mmin-mmin -10 Find any files modified in the last ten minutes or less -type-type f Find only regular files -user-user bob Find any files owned by the user bob

33. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. less Command

34. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Viewing files with less command The less command is a pager command designed to display only one page of data at a time. The more command is another pager command that has less features than the less command. Both commands allow the user to move back and forth with movement commands to view one page at a time.

35. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. The help screen in less Once in the less program, pressing the "h" key will display the help screen:

36. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. less movement commands As seen in the help screen, the less command has many movement commands. The most common commands are: MovementKey Window forwardSpacebar Window backwardb Line forwardEnter Exitq Helph

37. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. less searching commands Type / to search from cursor to end of file. Type ? to search from cursor to beginning of file. Type pattern to search and press Enter. If more than one match found, press n to go to next match or N to go to previous match.

38. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. head or tail

39. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Filtering with head The head command displays the first ten lines of a file by default. The -n option allows for the number of lines to be displayed to be specified.

40. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. head with negative lines Normally the head command displays the number of lines specified from the top of the file. Using -n with a negative value, indicates how many lines from the bottom to not show. This example shows all lines from /etc/passwd except the last thirty-two.

41. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Filtering with tail The tail command displays the last ten lines of a file by default. The -n option allows for the number of lines to be displayed to be specified:

42. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. tail with positive lines If the -n option specifies the number of lines to be displayed with a plus ( + ) prefix, then the tail command interprets that to mean to display from that line number to the end of the file:

43. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Following with tail The tail command is able to monitor changes a file and print them out as they occur by using -f option. System administrators frequently follow log files in order to troubleshoot system problems. The user must terminate the tail command when following with the –f option by using CTRL-C.

44. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. sort Command

45. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Sorting files or input The sort command will rearrange its output lines according to one or more fields you specify for sorting. Fields are separated by whitespace, although with the –t option, you can specify the delimiter. The default sort is in ascending order, but you can use the -r option to reverse the sorting of a field. The default sort is a dictionary sort, but you can use the -n option to make it a numeric sort.

46. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Example of sort In the following example, the /etc/passwd file is sorted using a : character as a delimiter, by the fourth field numerically and then the third field numerically in reverse:

47. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. File Statistics

48. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. File statistics with wc command The wc command outputs up to three statistics for each file it is given as an argument. By default, wc displays the lines, words and bytes contained in each file. If provided more than one file, then it also calculates the totals of all files. To view individual statistics, specify -l for lines, -w for words or -c for bytes.

49. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Example of wc command To analyze the number of lines, words and bytes in the /etc/passwd and /etc/passwd- files, the following wc command could be executed:

50. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Using wc with pipes The wc command is often used with pipes so that the output of a command can be analyzed. Using wc -l as the final command in the pipe will count how many lines of output was produced. For example, to determine how many files and directories are in the /etc directory, you could execute: ls /etc | wc -l

51. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. cut Command

52. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Filtering with cut command If you want to extract columns of text, then the cut command provides two simple techniques: – By delimiter, where whitespace is the default. The -d option can let you specify other delimiters and -f is used to indicate which fields to extract. – By character position, using the -c option with the range of the column to extract.

53. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Example of cut command The /etc/passwd file is delimited by colon with these fields: account:password:UID:GID:GECOS:directory:shell To extract the first, and fifth through seventh fields: cut –d: -f1,5-7 /etc/passwd

54. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. grep Command

55. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Filtering with grep command The grep command can be used to filter standard input or the contents of a file for lines matching a specified pattern. If you want to see where a pattern, or perhaps a word, appears within a file, then the grep command is useful for that purpose.

56. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Common grep options OptionPurpose --colorColor the matches found -vReverse (negate) matches -cCount matches -nNumber matching lines -lList matching files -iMatch case insensitive -wMatch pattern as a word

57. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Basic Regular Expressions

58. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Basic Regular Expressions Basic Regular Expressions (BRE) are able to be used with the grep command without requiring an option to use them (unlike Extended Regular Expression show later). The simplest regular expressions are just alphabetic or numeric characters that match themselves. The backslash \ can be used to escape the meaning of regular expression meta-characters, including the backslash itself.

59. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: the. example The. (period) character matches exactly one character. The example below shows the grep command matching the 'a' followed by two characters. The results show it matched 'abc'.

60. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: the [ ] example The [ ] (brackets) characters are used to match exactly one character. The characters can be listed or given as a range. If the first character listed is ^ (caret), then it means not the characters bracketed.

61. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: the * example The * (asterisk) character will match zero or more of the previous character. Matching "a*" is not very useful because it might match zero a's (matches every line). Matching "abcd*" would be more useful, since you would need an "abc" followed by zero or more d's.

62. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: the ^ example The ^ (caret) character, when appearing at the beginning of the pattern, means that pattern must appear at the beginning of the line. The ^ not at the beginning of a pattern matches itself.

63. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: the $ example The $ (dollar sign), when appearing at the end of the pattern, means that pattern must appear at the end of the line. The $ not at the beginning of a pattern matches itself.

64. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. BRE: Combining ^ and $ Combining both the ^ and $ characters allows for two special matches: – '^$' is a blank line match. – '^pattern$ matches if the whole line contains only the "pattern“.

65. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Extended Regular Expressions

66. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. Extended Regular Expressions The use of Extended Regular Expressions (ERE) requires the -E option when using the grep command. Extended Regular Expressions can be combined with Basic Regular Expressions. The following are ERE characters: ?, +, and |

67. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. ERE: the + example The + (plus) character will match one or more of the previous character. Matching "a+" is useful because it can match one or more a's, ensuring only lines that have at least one “a” are matched.

68. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. ERE: the ? example The ? (question mark) character will optionally match one of the previous character. The ? character is useful for matching characters that only occasionally appear in a word. The following example illustrates this:

69. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. ERE: the | example The | (vertical bar) character will act like an “or” operator between two regular expressions. This alternation operator is useful to be able to match multiple patterns:

70. This slide deck is for LPI Academy instructors to use for lectures for LPI Academy courses. ©Copyright Network Development Group 2013. The xargs command The xargs command helps complex piped command sets execute more efficiently It attempts to build the longest command line possible with as many arguments as possible It tries to prevent executing the command each time for every argument