23penguins has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to automate a website download. The website is https address and is a password protected site. I have used lwp in the past to capture html pages, mostly by just copying code and modifying it. This is different. By using liveheaders I found out the url I am looking for is "https://mysite.com/myapp.dll<someotherstuffhere>" When this is typed in a browser window, the download browser window comes up.
Live headers has this "Content-Disposition: attachment; filename="myfilename.tgz" Transfer-Encoding: chunked Content-Type: application/octet-stream"
Here is the code I used before :

 my $ua = LWP::UserAgent->new();
    $ua->agent("USER/AGENT/IDENTIFICATION");
# make request
    my $request = HTTP::Request->new(GET => $URI);
# authenticate
    $request->authorization_basic($user, $pass);
# accept response
    my $response = $ua->request($request);

[download]
Now I get as far as making the request and it hangs..
I am using ActiveState Perl 5.10.1
Question : can I use lwp to download these files ( I have hundreds of them ) or should I approach this another way?
thanks in advance

Replies are listed 'Best First'.
Re: web downloading via lwp
by cormanaz (Deacon) on May 19, 2010 at 00:32 UTC
    This is not a lot of info to go on. But you might have a look at Win32::IEAutomation. Sometimes it's the only thing you can use if you want to access sites that have javascript, applets, flash, and such like. It basically hacks IE to let your perl script do the driving.

    My guess is that you could use this to open a download window in IE and download the file, then you'd have to find and open that file with your script.

    Of course you have to be on windoze to use it.

[reply]
      Sorry for the confusion. All the lwp examples I have seen return content. And when they talk about saving files, they talk about saving the content of the request. Basically , my url does not point to a static text page, or a file. My url runs code. The code finds the file, pulls it out of the repository, packages it up and then launches the "save as" dialog box. I have looked at Win32::IEAutomation. I cannot figure out how to automate the "open a download window in IE and download the file" part. My url already already opens the download window. Thanks
[reply]
Re: web downloading via lwp
by Gangabass (Vicar) on May 19, 2010 at 01:43 UTC

    What you mean by "password protected"? May be you're talking about login web form which you must fill first? When you can try WWW::Mechanize.

[reply]
Re: web downloading via lwp
by Khen1950fx (Canon) on May 19, 2010 at 08:16 UTC
    It seems that you'll need to say which protocol that you want to use. Since the website is https, then add that to the script like I've done here:
    #!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->env_proxy;
$ua->protocols_allowed(['https']);

my $URI = 'https://www.somemonks.org';
my $request = HTTP::Request->new(GET => $URI);
$request->authorization_basic('user', 'pass');

my $response = $ua->request($request);
if ($response->is_success) {
    print $response->decoded_content;
}
else {
    die $response->status_line;
}

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom