Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Download file over HTTP

by l3nz (Friar)
on Nov 25, 2003 at 10:04 UTC ( [id://309846]=note: print w/replies, xml ) Need Help??


in reply to Download file over HTTP

Frankly, I don't see much of a problem with LWP, simply use the content method and not as_string.

The piece of code belows downloads a file and compares it with itself using the SHA algorithm in order to prove the download was correct. I tested it with a number of binary files that were available on my Windows box and always worked right.

use strict; use LWP::UserAgent; use Digest::SHA1; my $srcfile = "c:/...../cab/WBCust.CAB"; my $tempdl = "temp.tmp"; my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET => 'http://127.0.0.1/...../cab/WBCust +.CAB'); my $r = $ua->request($req)->content; open F, ">$tempdl" or die "$! $tempdl"; binmode F; print F $r; close F; computeSHA( $srcfile ); computeSHA( $tempdl ); unlink $tempdl; exit; sub computeSHA() { my ($srcfile) = @_; my $sh_pre = Digest::SHA1->new; open S1, $srcfile or die "$! $srcfile"; $sh_pre->addfile(*S1); close S1; print "$srcfile:\n " . $sh_pre->hexdigest . "\n"; }
The result is something like:
c:/...../cab/WBCust.CAB: 3816f34e861b6268c716b9f06091dee0d580f2cb temp.tmp: 3816f34e861b6268c716b9f06091dee0d580f2cb

Replies are listed 'Best First'.
Re: Re: Download file over HTTP
by chaskins (Sexton) on Nov 25, 2003 at 11:29 UTC
    Thanks everyone for you help, you've all given me very useful advice!

    Thanks
    Chris
Re: Re: Download file over HTTP
by chaskins (Sexton) on Nov 25, 2003 at 11:58 UTC
    OK it looks like I'll be downloading a 45MB file, is LWP ok/the right choice to be downloading this file?

    Thanks

    Chris
      What if it drops the connection halfway through on a slower link? You will have to cater for that, like resuming a download at a given position, but not all servers allow that.

      I'd say it's best to yourself a download manager, like getright, netant, etc, that are specialized in downloading large files efficiently from the Internet. They can do special speed optimization too, like openning multiple connections and download sections of the file simutaneously.

        What I'm trying to do is automate the downloading of the backupfile created from my web site. You need to login, auth done via .htaccess, and then you have access to download the file over http. I wanted to write a perl script which I could get to run once a day, via cron, to do this. Your right I could use getright etc but these packages don't run on the command line, and with no interaction to enable them to run as a cron job.

        Thanks

        Chris

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://309846]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2024-04-24 22:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found