Uniq

The uniq command is able to isolate all of the unique lines from the text file. You can pipe the contents of the text file into uniq.

cat textfile.txt | uniq

but this only works if the duplicate lines are adjacent to each other. In order for the lines to be adjacent, they would first need to be sorted into alphabetical order.

sort textfile.txt | uniq

Working example:

sort playeridsduplicates.txt|uniq>playerids.txt

To see how many occurrences of each line is in the file, we can use the -c (count) option with uniq.

sort textfile.txt | uniq -c

If we need the text file to retain its previous order

awk '!seen[$0]++' textfile.txt

The first occurrence of a line is kept, and future duplicate lines are suppressed.

Working example:

awk '!seen[$0]++' playeridsduplicates.txt > playerids.txt