in reply to Using hashes or arrays to remove duplicate entries
Why not create a unique key at the database level, then when you try to
insert an existing value it database will disallow it?
Trap this in your program.
This method will let you keep distinct values between runs as well as inside runs.
Better create a hash as suggested to prevent attempts to insert within the run.