Hash, definitely. I wouldn't even think about any other alternative, unless and until the memory consumption becomes an issue. Even then, there are probably tied hash classes that store their data on disk rather than in memory.
Comment on Re: Bloom Filter or other mehod to store URL's?
Hmmm... our intranet server hosts about 10 million pages. If i do a no-brain hash, it quickly fills my 1 GB of mem. Even on disk a 10 GB hash file doesn't sound nice.