in reply to Re^2: Proxy with mechanize
in thread Proxy with mechanize
I have tidied up the code a bit... and changed your proxy line (by adding 'http://' to the proxy address). This now works from behind my proxy server and I guess is the solution to your problem.use LWP::Debug qw(+);
When you print $links_to_check you will get a response like WWW::Mechanize::Link=ARRAY(0x2a01214), this is because you are printing a Mechanize Link object (see WWW::Mechanize::Link), you can dig down a bit to get the url etc and the documentation should help you with that.#!/usr/bin/perl use strict; use warnings; use Data::Dumper; use LWP::UserAgent; use LWP::Debug qw(+); use WWW::Mechanize; my $total_count = 0; my @keywords = ('simpsons', 'diehard', 'fantastic+four', ); foreach my $keywords(@keywords){ my $url ='http://news.google.co.uk/search?q=~%22'.$keywords.'%22&num +=100&hl=en&safe=off&start=0&as_qdr=all&filter=0'; my $mech = WWW::Mechanize->new(); $mech->proxy('http','http://127.0.0.1:8088'); $mech->get($url); print $mech->uri."\n"; my @links_to_check = grep { $_->url() !~ /google/i} $mech->find_a +ll_links( url_regex => qr/\./i ); foreach my $links_to_check (@links_to_check){ $total_count++; print "$links_to_check \n"; } } print " $total_count news items found \n";
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^4: Proxy with mechanize
by coder57 (Novice) on Aug 02, 2007 at 19:54 UTC |