A check on whether pages on a web site have changed

Win has asked for the wisdom of the Perl Monks concerning the following question:

Please could anyone suggest the best coded method I can use to create a program that will check to see if any web pages on a web site have changed. Is there free software available to me for this or am I best of coding something in Perl?

Comment on A check on whether pages on a web site have changed

Replies are listed 'Best First'.
Re: A check on whether pages on a web site have changed by davorg (Chancellor) on Nov 16, 2005 at 11:37 UTC
LWP::Simple contains a "mirror" function that does something similar. The description says: mirror($url, $file) Get and store a document identified by a URL, using If-modified-since, and checking the Content-Length. Returns the HTTP response code. You could either use that function or use the code as a basis for your own program. -- <http://dave.org.uk> "The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg	[reply]
Re: A check on whether pages on a web site have changed by spatterson (Pilgrim) on Nov 16, 2005 at 11:52 UTC
You might find some of the comments on How to get web creation date from webserver? useful, especially as servers don't always return a date modofied header. just another cpan module author	[reply]
Re: A check on whether pages on a web site have changed by marto (Cardinal) on Nov 16, 2005 at 12:08 UTC
Win, "Is there free software available to me for this" If you have a local copy of the site you could use rsync with its --list-only option to display (or log) files that have changed. Hope this helps. Martin	[reply]