in reply to Mirroring Script

The dirtiest way to do this is: 

#!/usr/bin/perl -w

use strict;

for (1..273) {
    `wget http://www.my-site.com/mydir/?Page=$_`;
}

[download]

Note: this code is untested, and is missing any sort of error checking and assumes you're using a UNIX like system with the wget command installed. Not much of a solution.
Do not trust this code, but you get the idea.

A better way would be to have a look at the modules and utilities in Bundle::LWP and base a more robust script on those.

Implementation is left as a task for the reader.

Update: Beaten to it by moodster! :-) Great minds think alike. ;)

Further update: I feel quite honored to have merlyn's attention :-)
I posted the incorrect/dirty/horrific snippet as a basis of one solution, LWP::Simple is clearly a better solution - and I made a point of mentioning Bundle::LWP, which includes the mirror functionality that merlyn has demostrated.

So, ignoring the obvious benefits of LWP::Simple, I have seen the error of my ways with the use of backticks here - system() should have been used instead of backticks to fork wget.
That method still isn't fanstatic - I stand by my disclaimer :-)

Replies are listed 'Best First'.
Re: Re: Mirroring Script
by merlyn (Sage) on Feb 05, 2002 at 15:28 UTC
    The dirtiest way to do this is: 
    #!/usr/bin/perl -w 
 
use strict; 
 
for (1..273) { 
    `wget http://www.my-site.com/mydir/?Page=$_`; 
}

    [download]
    Ouch! You're not kidding. Using backquotes in a void context! Not only is it messy because it forks needlessly (thanks to the question mark), but you're also capturing the output just to discard it!

    Here's code that will be much saner and faster. In fact, you can run it multiple times, and it downloads only the changed files, if "if-modified-since" is supported by the server: 

    use LWP::Simple qw(mirror);
for (1..273) {
  mirror "http://www.my-site.com/mydir/?Page=$_", "file$_";
}

    [download]

    -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
[select]

      Note: this code is untested, and is missing any sort of error checking and assumes you're using a UNIX like system with the wget command installed. Not much of a solution.
      Do not trust this code, but you get the idea.

      I think this disclaimer shows how little faith I put in that snippet :-)

      Certainly LWP::Simple is the way to go. merlyn++

[reply]
Re: Re: Mirroring Script
by gellyfish (Monsignor) on Feb 05, 2002 at 14:50 UTC

    Infact LWP includes an lwp-mirror that may well do what is wanted

    /J\

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom