I needed a general solution to this sort of problem, where I might need to eliminate lines with duplicate values in column 3 of one file, column 8 in some other file, and the combination of columns 2 and 6 in yet another file. I also wanted to choose between keeping only the first line containing duplicated content, vs. keeping only the last line. I just posted it here at the Monastery: col-uniq -- remove lines that match on selected column(s)