in reply to finding and deleting repeated lines in a file
It sounds like a quick hash check keyed on the sequence itself. If the data source is large, it'll need a tie and maybe some MD5 action.
-- Steve Marvell