Re^3: Best way to recursively grab a website

LWP ships with a number of example applications / utilities such as GET, POST etc. One of these utilities is lwp-mirror.

From the documentation:

This program can be used to mirror a document from a WWW server. The document is only transfered if the remote copy is newer than the local copy. If the local copy is newer nothing happens.

Comment on Re^3: Best way to recursively grab a website

Replies are listed 'Best First'.
Re: Best way to recursively grab a website by b10m (Vicar) on Mar 29, 2005 at 12:26 UTC
Or lwp-rget This program will retrieve a document and store it in a local file. It will follow any links found in the document and store these documents as well, patching links so that they refer to these local copies. This process continues until there are no more unvisited links or the process is stopped by the one or more of the limits which can be controlled by the command line arguments. -- b10m All code is usually tested, but rarely trusted.	[reply]

Replies are listed 'Best First'.

Re: Best way to recursively grab a website
by b10m (Vicar) on Mar 29, 2005 at 12:26 UTC

Or lwp-rget

This program will retrieve a document and store it in a local file. It will follow any links found in the document and store these documents as well, patching links so that they refer to these local copies. This process continues until there are no more unvisited links or the process is stopped by the one or more of the limits which can be controlled by the command line arguments.

--
b10m

All code is usually tested, but rarely trusted.

[reply]