Gtforce has asked for the wisdom of the Perl Monks concerning the following question:

Am new to perl and the monastery. Am looking to download a zip file - the download itself happens, but I am unable to unzip the file after download. I suspect I'm doing something wrong with the way I'm downloading the zip file (encoding et al). The error I get when unzipping it - "254 extra bytes at beginning or within zip file". Would appreciate some guidance. 

use LWP::UserAgent;
  
  my $nseurl = "https://nse-india.com/content/historical/EQUITIES/2017
+/AUG/cm14AUG2017bhav.csv.zip";
  my $file = "08SEP2017bhav.csv.zip";
  my @rows; 
  my $ua = LWP::UserAgent->new;
  $ua->agent("MyApp/0.1 ");

  my $req = HTTP::Request->new(POST => $nseurl);
  $req->content_type('X-UA-Compatible" content="IE=8');
  $req->content('text/csv; charset=utf-8');

  my $res = $ua->request($req);
  @rows = $res->content;
  
  open ( my $fhwrite, ">>", "$file") or die "$file: $!";
  if (defined $fhwrite) {
    print $fhwrite @rows;
    close ( $fhwrite );
 }

[download]

Replies are listed 'Best First'.
Re: LWP::UserAgent downloading zip files
by 1nickt (Canon) on Sep 10, 2017 at 16:28 UTC

    Hi, welcome to the Monastery.

    Your code seems to have a number of problems.

    • You are trying to fetch a file but are using a POST request. The server seems to expect a GET.
    • You are using content() to apparently try to set a header (content-type?).
    • You are trying to set the content-type to CSV but you are requesting a zip file.

    Here's how I would do it (using the more modern, lighter-weight, simpler HTTP::Tiny to make the http request): 

    use strict; use warnings;
use HTTP::Tiny;
use IO::Uncompress::Unzip qw(unzip $UnzipError);

my $url = 'https://nse-india.com/content/historical/EQUITIES/2017/AUG/
+cm14AUG2017bhav.csv.zip';
my $filename = '/path/to/your/folder/cm14AUG2017bhav.csv';

my $res = HTTP::Tiny->new->get( $url );
die 'Download failed' if not $res->{'success'};
unzip \$res->{'content'}, $filename or die "unzip died: $UnzipError";

__END__

    [download]

    After you get your CSV file successfully onto disk, make sure to use a proper module for reading it, such as Text::CSV.

    Hope this helps!


    The way forward always starts with a minimal test.
[reply]
[d/l]
[select]

      Funnily enough, the POST works just fine and gets me the file I'm looking for. I've seen other snippets of code all over the place and using POST to download files. I've got to sit down and look at the difference between these two. Thanks, 1nickt.

[reply]

        You are welcome. Trust me, there is a difference between GET and POST requests. It is quite possible that a server could be configured to accept GET and POST requests for the same URL, but your program should not assume that.

        As far as seeing snippets of code: Try to stick to the official documentation for the modules you use (and a good place to look for implementation examples is in the module distribution's tests) and build up your code following that so you know what you are doing. Copying chunks of code you spot in various places in the wild can be useful but should be accompanied with working hard to understand how the tools you are using really work.


        The way forward always starts with a minimal test.
[reply]
Re: LWP::UserAgent downloading zip files
by Anonymous Monk on Sep 09, 2017 at 21:35 UTC
    Try open ( my $fhwrite, ">>:raw", "$file") or die "$file: $!";
[reply]
[d/l]

      Perfect, many thanks.

[reply]
Re: LWP::UserAgent downloading zip files
by Anonymous Monk on Sep 09, 2017 at 22:49 UTC
    $ua->post( $url, ':content_file' => $filename, );
[reply]

Back to Seekers of Perl Wisdom