Re: How to remove duplicates from a large set of keys

Put the keys in a file, sort them by key at the file level (there are any number of tools to do this). Scan the file, any duplicates will be adjacent so just skip them as you scan. You may even find that the sort program will remove the dupes for you.

---
demerphq

Comment on Re: How to remove duplicates from a large set of keys

Replies are listed 'Best First'.
Re^2: How to remove duplicates from a large set of keys by Anonymous Monk on Feb 10, 2005 at 10:00 UTC
Please read the entire posting before shooting of a reply. Had you bothered to read the entire first paragraph (I know, I know, it's three lines, waaaaaaay too long), you could have read: In real time I should check does a new value exist in my set and if not to add it. Resorting a million record file each time a record in added isn't very efficient.	[reply]