Difference between revisions of "Uniq"
From Free Knowledge Base- The DUCK Project: information for everyone
(Created page with "The uniq command is able to isolate all of the unique lines from the text file. You can pipe the contents of the text file into uniq. cat textfile.txt | uniq but this only...") |
(No difference)
|
Latest revision as of 16:24, 10 February 2022
The uniq command is able to isolate all of the unique lines from the text file. You can pipe the contents of the text file into uniq.
cat textfile.txt | uniq
but this only works if the duplicate lines are adjacent to each other. In order for the lines to be adjacent, they would first need to be sorted into alphabetical order.
sort textfile.txt | uniq
Working example:
sort playeridsduplicates.txt|uniq>playerids.txt
To see how many occurrences of each line is in the file, we can use the -c (count) option with uniq.
sort textfile.txt | uniq -c
If we need the text file to retain its previous order
awk '!seen[$0]++' textfile.txt
The first occurrence of a line is kept, and future duplicate lines are suppressed.
Working example:
awk '!seen[$0]++' playeridsduplicates.txt > playerids.txt
