If they are duplicates, how do you tell the 1^st from the 2^nd one?

Anyway, if you use only a part of a line to find the duplicates, it makes sense. You can either reverse the input file and use the old algorithm, or you have to remember the last line seen for each key in a hash. The next problem of the latter is to get the original order of the lines.