in reply to Re^2: Normalizing URLs
in thread Normalizing URLs
What about using the "last_modified" method in LWP? Keep track of it locally. When you access the page again, check the time it was modified and skip it if that time is not newer than what you've saved.
This idea is from "Spidering Hacks" (hack #16).
|
|---|