in reply to removing duplicate lines
# assuming pre-sorted as in OP: uniq file1 > file2 [download]
use Data::Dumper; print Dumper \%uniquelines; [download]