Thanks for the thought. A hashmap of already-visited sites might work, but I am doing this from mod_perl2, which uses multiple threads for handling HTTP requests. Mod_perl2 seems to magically make each thread hold its own set of variables, and I have not yet figured out how to have a hashmap that is shared across all threads. And if I did figure out how to have a shared hashmap I would then need use some kind of locking to ensure thread safety when modifying that hashmap. So although the idea is intuitively appealing, I am concerned that it may not be so easy to implement.