pccode has asked for the wisdom of the Perl Monks concerning the following question:

I need to use the lwp-download program to download multiple files. I took the original lwp-download program included with perl and turned it into a subroutine. I did encounter one error though when I first ran the program. "Variable $shown will not stay shared." This error is referring to the shown variable in the show subroutine at the end of the program. I added a "my" to the shown variable and the error went away. 
my $shown++;

[download]
However, I don't know if that will create any problems with the rest of the program. To save space on this page I'm only going to post the code that I've added to the lwp-program. You can download the complete program at http://parasoft.netfirms.com/. 
use warnings;
use strict;

# Enter the local directory to save downloaded files into
my $savelocation = "c:\\docume~1\\name\\desktop\\downloads\\";

# If directory does not exist, create it
opendir(FINDDIR, $savelocation) || mkdir($savelocation);
closedir FINDDIR;

# Use the subroutine below for each file you need to download
&getFile("http://www.domainname.com/file.png");
&getFile("http://www.domainname.com/file.txt");


sub getFile {
     my $netlocation = $_[0];
     my @directories = split(/\//, $netlocation);
     my $fname = pop(@directories);
     &Download($netlocation, "$savelocation$fname");
}

sub Download {
# the entire lwp-download program
}

[download]
There are two problems with the program. It only executes the first getFile subroutine before it exits and it overwrites existing files without prompting. How can I fix this? Thanks.

Replies are listed 'Best First'.
Re: lwp-download program
by holli (Abbot) on May 29, 2005 at 19:59 UTC
    No need to take the original lwp-download program included with perl and turn it into a subroutine

    Simply use the getstore($url, $file) function of the LWP::Simple module.


    holli, /regexed monk/
[reply]
[d/l]
[select]
      Holli, your suggestion works perfectly, but I was using the lwp-download program since it was designed for downloading larger files. It's nice to be able to see the status and time remaining for each download. Thanks.
[reply]
Re: lwp-download program
by Zaxo (Archbishop) on May 29, 2005 at 20:07 UTC

    There are two basic ways to check whether a file exists before clobbering it. You can use stat or its little cousin -e on the filename before opening, or else you can call sysopen with the O_EXCL flag set and catch the error when the file exists.

    As near as I can see, lwp-download is a one-shot program not designed for downloading multiple files. In your wrapper, you could accept a list of urls from the command line and loop through calls to lwp-download. I don't know why you are getting only one getFile() call through unless you are hitting some fatal error in the first call.

    Why don't you just call lwp-download through system instead of pasting it into a subroutine? I'm very suspicious of that procedure, there are lots of traps you can trip doing that.

    After Compline,
    Zaxo

[reply]
      The lwp-download script actually checks for existing files and then prompts you before it overwrites them. Unfortunately that part of the program stopped working in the download subroutine. 
      # Check if the file is already present
          if (-f $file && -t) {
          $shown = 1;
          print "Overwrite $file? [y] ";
          my $ans = <STDIN>;
          unless (defined($ans) && $ans =~ /^y?\n/)                   
+           {
              if (defined $ans) {
              print "Ok, aborting.\n";
              }
              else {
              print "\nAborting.\n";
              }
              exit 1;
          }
          $shown = 0;
          }
          else {
          print "Saving to '$file'...\n";
          }

      [download]
      I will try your suggestion to use system instead of the subroutine.
[reply]
[d/l]
        The system function solves the problem. Thanks.
[reply]

Back to Seekers of Perl Wisdom