in reply to downloading images from a webpage

You don't call getstore you call get.

Replies are listed 'Best First'.
Re^2: downloading images from a webpage
by Aldebaran (Curate) on Apr 07, 2012 at 18:33 UTC

    that was my first tack, and I thought I was on the right track, but what I ended up with using getstore() was files that kind of thought they were jpg's and kind of thought they were html docs. Here's the script I used:

     
#!/usr/bin/perl -w
use strict;


use LWP::Simple;  

open FILE, "text1.txt" or die $!;


my $url;
my $text;

while (<FILE>)
{
$text = $_;

$url = 'http://www.nobeliefs.com/nazis/' . $text;
$text =~ s#images/##;
print "$url\n";
print "$text\n";

getstore($url, $text) or die "Can't download: $@\n";

}

    [download]

    an ls command shows question marks: 

    $ ls
...
prayingHitler.jpg?
PraysingCelebration.jpg?
priests-salute.jpg?
received.jpg
reichchurch.gif?
...

    [download]

    and when I open up a jpg it looks like this:

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http:/
+/www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html>

<head>

<meta http-equiv="Content-type" content="text/html; charset=utf-8">

<title>Website Moved</title>

<style type="text/css">

.statusBox {

    width: 80px;

}

.fb {

    width:43%;

    float:left;

    text-align:center;

    margin:5px 20px 5px 20px;

    padding:20px 0 20px 0px;

    background:#eef8fd;

    height:110px;

    border:solid 1px #dff4fe;

}

.fb2 {

    width:43%;

    float:right;

    text-align:center;

    margin:5px 20px 5px 20px;

    padding:20px 0 20px 0px;

    background:#eef8fd;

    height:110px;

    border:solid 1px #dff4fe;
...

    [download]

    I think the trick might be to find a way to define $params such that this works, but I haven't been able to do that yet. (I only get errors)

    my $data = LWP::Simple::get $params{URL};
my $filename = "image.jpg";
open (FH, ">$filename");
binmode (FH);
print FH $data;
close (FH);

    [download]
[reply]
[d/l]
[select]

      I finally got output, but it looks like a kid did it. I'd like to polish it up and be able to have a script that uses WWW::Mechanize more effectively.

      #!/usr/bin/perl -w
use strict;
use LWP::Simple;  
open FILE, "text1.txt" or die $!;
my $url;
my $text;
while (<FILE>)
{
$text = $_;
$text =~ s/\s+//;
$url = 'http://www.nobeliefs.com/' . $text;
print qq[ '$url' ];
$text =~ s#images/##;
print "$text\n";
getstore($url, $text) or die "Can't download: $@\n";
}

      [download]

      How would I use chomp instead of $text =~ s/\s+//;? Nothing I tried worked.

      My failure with WWW::Mechanize was almost complete. The most I could get it to do was dump the names of the images to STDOUT. How could I re-write this to avoid all the nonsense with saving to a file which I then have to read? The syntax for $mech->images is: Lists all the images on the current page. Each image is a WWW::Mechanize::Image object. In list context, returns a list of all images. In scalar context, returns an array reference of all images. I tried a dozen different things, but I don't get why this is not list context: 

      #!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
open FILE, "text2.txt" or die $!;
my $domain = 'http://www.nobeliefs.com/nazis.htm';
my $m = WWW::Mechanize->new;
$m->get( $domain);
my @list = $m->images();
print "@list \n";
#$m->text();
#$m->content( format => 'text2.txt' );
#print FILE $m;
close FILE;

      [download]
[reply]
[d/l]
[select]
        Hi. Sorry for the delay.

        You do call images() in list context. What you're going to want to do is combine this with getstore to save your images: 

        my @list = $m->images();
for my $img (@list) {
  my $url = $img->url_abs();
  my $filename = ...;  (fill this in, probably based on $url)
  getstore($url,$filename) or die "Can't download '$url': $@\n";
}

        [download]
[reply]
[d/l]

        I really screwed up this thread with a lot of newbie mistakes, but I wanted to thank blake for helping me combine a couple of the things I've been trying to do here. I can't seem to do so where his reply was, but I would like him to see that his script improvements worked. Let me also state that I'm as far from national socialism ideologically as I could be: I support civil rights, union rights, democracy, and I'm a pacifist.

        $ perl hitler8.pl
$ cat hitler8.pl
#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use LWP::Simple;
my $domain = 'http://www.nobeliefs.com/nazis.htm';
my $m = WWW::Mechanize->new;
$m->get( $domain);
my $counter = 0;
my @list = $m->images();
for my $img (@list) {
  my $url = $img->url_abs();
  $counter++;
  my $filename = "site/image_". "$counter";
  getstore($url,$filename) or die "Can't download '$url': $@\n";
}

$ cd site/
$ ls
image_1   image_15  image_20  image_26  image_31  image_37  image_42  
+image_9
image_10  image_16  image_21  image_27  image_32  image_38  image_43
image_11  image_17  image_22  image_28  image_33  image_39  image_5
image_12  image_18  image_23  image_29  image_34  image_4   image_6
image_13  image_19  image_24  image_3   image_35  image_40  image_7
image_14  image_2   image_25  image_30  image_36  image_41  image_8
$ echo "thx all for comments"
thx all for comments
$

        [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom