uniq The uniq command is useful when you need to find duplicate lines in a file. The basic format of the command is uniq in_file out_file In this format, uniq copies in_file to out_file, removing any duplicate lines in the process. uniq's definition of duplicated lines are consecutive-occurring lines that match exactly. If out_file is not specified, the results will be written to standard output. If in_file is also not specified,uniq acts as a filter and reads its input from standard input.

$ cat names Charlie Tony Emanuel Lucy Ralph Fred Tony $ uniq names Print unique lines Charlie Tony Emanuel Lucy Ralph Fred Tony

Tony still appears twice in the preceding output because the multiple occurrences are not consecutive in the file, and thus uniq's definition of duplicate is not satisfied. To remedy this situation, sort is often used to get the duplicate lines adjacent to each other. The result of the sort is then run through uniq. $ sort names | uniq Charlie Emanuel Fred Lucy Ralph Tony So the sort moves the two Tony lines together, and then uniq filters out the duplicate line

The -d Option Frequently, you'll be interested in finding the duplicate entries in a file. The -d option to uniq should be used for such purposes: It tells uniq to write only the duplicated lines to out_file (or standard output). Such lines are written just once, no matter how many consecutive occurrences there are. $ sort names | uniq -d List duplicate lines Tony

-c option The -c option to uniq behaves like uniq with no options (that is, duplicate lines are removed), except that each output line gets preceded by a count of the number of times the line occurred in the input. $ sort names | uniq –c Count line occurrences 1 Charlie 1 Emanuel 1 Fred 1 Lucy 1 Ralph 2 Tony