Re^2: Normalizing URLs

Some background - I am developing a scraper and it needs to know if it has scraped the page already - hence the normalization. It needn't be perfect, just good enough for all but the most arcane. Needs only work with http.

Comment on Re^2: Normalizing URLs

Replies are listed 'Best First'.
Re^3: Normalizing URLs by spiritway (Vicar) on Jul 22, 2005 at 03:18 UTC
What about using the "last_modified" method in LWP? Keep track of it locally. When you access the page again, check the time it was modified and skip it if that time is not newer than what you've saved. This idea is from "Spidering Hacks" (hack #16).	[reply]


laziness, impatience, and hubris
	PerlMonks