Re^4: Anyone know why I can't scrape this page?

Basically the program is going to pull the line movement data from the website. I have another portion of the program which harvests the links off the site and puts them into an array. For example, if I'm looking at NFL odd in vegas, it will go to the nfl odds in vegas page, and pull all the links to the games there.

From this array I want to use a subroutine (using WWW::Mechanize) to go to the links in the array and download the page to my computer.

I think the problem I have stems from the fact that there is an @ sign in the url. When I test the script with a variable with single quotes, I'm able to get the page. When I try to use the subroutine I get an error. 400 bad request

Comment on Re^4: Anyone know why I can't scrape this page?

Replies are listed 'Best First'.
Re^5: Anyone know why I can't scrape this page? by Lawliet (Curate) on Sep 07, 2008 at 18:54 UTC
That is odd. Your code works fine when I simply ask to show the content. `#!/usr/bin/perl -w use strict; use WWW::Mechanize; my $mech = WWW::Mechanize->new; my $url = 'http://www.vegasinsider.com/nfl/odds/las-vegas/line-movemen +t/bengals-@-ravens.cfm/date/9-07-08/time/1300#J'; save_file($url); sub save_file { my $mech = WWW::Mechanize->new; $mech->timeout(60); my @game = @_; foreach(@game) { print "$_\n"; $mech->get($url) or die "Can't get url"; print $mech->content; } }` [download] As long as you initially quote the url in single quotes it should not interpolate no matter how it is used. As long as I cannot reproduce the error I cannot fully help you to diagnose the problem. There is one thing that I feel obligated to bring to the table that I think you may have missed: NOTE: Because :content_file causes the page contents to be stored in a file instead of the response object, some Mech functions that expect it to be there won't work as expected. Use with caution. Unfortunately, that does not answer why your code works outside a subroutine. I'm so adjective, I verb nouns! chomp; # nom nom nom	[reply] [d/l]

Replies are listed 'Best First'.

Re^5: Anyone know why I can't scrape this page?
by Lawliet (Curate) on Sep 07, 2008 at 18:54 UTC

That is odd. Your code works fine when I simply ask to show the content.

#!/usr/bin/perl -w
use strict;

use WWW::Mechanize;

my $mech = WWW::Mechanize->new;
my $url = 'http://www.vegasinsider.com/nfl/odds/las-vegas/line-movemen
+t/bengals-@-ravens.cfm/date/9-07-08/time/1300#J';

save_file($url);

sub save_file {
    my $mech = WWW::Mechanize->new;
    $mech->timeout(60);

    my @game = @_;
    
    foreach(@game) {
        print "$_\n";
        $mech->get($url) or die "Can't get url";
        print $mech->content;
    }
}
[download]

As long as you initially quote the url in single quotes it should not interpolate no matter how it is used. As long as I cannot reproduce the error I cannot fully help you to diagnose the problem.

There is one thing that I feel obligated to bring to the table that I think you may have missed:

NOTE: Because :content_file causes the page contents to be stored in a file instead of the response object, some Mech functions that expect it to be there won't work as expected. Use with caution.

Unfortunately, that does not answer why your code works outside a subroutine.

I'm so adjective, I verb nouns!

chomp; # nom nom nom

[reply]
[d/l]