Download a file from website

rahulruns has asked for the wisdom of the Perl Monks concerning the following question:

Need to download a file from a website I have tried using LWP::Simple and curl but the connection closes before the entire file is downloaded. The file size is around 20MB and only around 6MB is downloaded

LWP::Simple code

use LWP::Simple;
my $url  = 'https://cwiki.apache.org/confluence/display/PIG/PigTutoria
+l';
my $file = 'pigtutorial.tar.gz';
my $status = getstore($url, $file);
die "Error $status on $url" unless is_success($status);

CURL code

my $url = `curl -C - -o pigtutorial.tar.gz https://cwiki.apache.org/co
+nfluence/display/PIG/PigTutorial`;
[download]

Comment on Download a file from website Download Code

Replies are listed 'Best First'.
Re: Download a file from website by frozenwithjoy (Priest) on Sep 13, 2012 at 08:06 UTC
Does it work if you try to do it outside of Perl? When I run your curl command in terminal (`curl -C - -o pigtutorial.tar.gz https://cwiki.apache.org/confluence/display/PIG/PigTutorial`), I just get a 65kb file that is the html of the site (and not a tarball). Are you pointing to the correct place?	[reply] [d/l]
Re^2: Download a file from website by rahulruns (Scribe) on Sep 13, 2012 at 08:10 UTC
Yes it works outside PERL. I am able to download the file and extract it	[reply]
Re^2: Download a file from website by rahulruns (Scribe) on Sep 13, 2012 at 10:23 UTC
rightly replied by frozenwith joy. When you check the file downloaded file from the link it has a href for the location of exact file and now it downloads from that location mentioned in href. This thread could be closed	[reply]
Re: Download a file from website by Anonymous Monk on Sep 13, 2012 at 07:54 UTC
If the remote server is timing you out, there is nothing you can do, esp if it doesn't support resuming wget -c -O filename -- http... sub LWP::UserAgent::resume	[reply]
Re^2: Download a file from website by rahulruns (Scribe) on Sep 13, 2012 at 08:08 UTC
I went through the code provide for LWP::UserAgent but it is taking time to make out where do I need to change the specific URL and filename. Could you help on that part	[reply]