in reply to Re^6: Can't call method "proxy" on an undefined value at
in thread Can't call method "proxy" on an undefined value at

You might find it cleared using the same approach for all your loops and print statement to check what is happening. Note I have added a number to the output filename to create a different file for each proxy. 

#!perl
use strict;
use LWP::UserAgent;
use File::Path 'make_path';

my $link1='';
my $link2='';
my $link3='';

# make directories
my $dir = 'Bing/1Parsed/Html/';
unless (-d $dir){
  make_path($dir, { verbose=>1, mode=>0755 });
}

# keywords
open my $fh, '<', 'keywords.txt' or die $!;
chomp(my @keywords = <$fh>);
close $fh;

# cities
open my $fh, '<', 'cities.txt' or die $!;
chomp(my @cities = <$fh>);
close $fh;

# proxies
open my $fh, '<', 'proxies.txt' or die $!;
chomp(my @proxies = <$fh>);
close $fh;
  
# create useragent
my $ua = LWP::UserAgent->new; 
$ua->agent('Mozilla/8.0');

# get pages for each city, keyword, proxy 
for my $city (@cities) {
  for my $keyword (@keywords){
    print "City :    [$city]\nKeyword: [$keyword]\n";

    for my $i (0..$#proxies){
      print "Current proxy:$proxies[$i]\n";

      # Use this UA/Proxy to fetch something....
      $ua->proxy(['http'], 'http://'.$proxies[$i]);     
      my $url = join '',$link1 . $city ."+". $keyword,
                        $link2 . $city ."+". $keyword,
                        $link3;
      my $response = $ua->get($url);
      print "getting $url\n";      
      if ($response->is_success) {
        my $filename = "Bing/${city}_${keyword}_${i}.html";
        print "Creating $filename\n";
        open my $fh, ">", $filename 
             or die("Could not open $filename. $!");
        print $fh $response->decoded_content;  # or whatever
        close $fh;
      } else {
        die $response->status_line;
      }
      print "\n"; 
    }

  }
}

[download]
poj

Replies are listed 'Best First'.
Re^8: Can't call method "proxy" on an undefined value at
by Anonymous Monk on Feb 09, 2016 at 14:22 UTC

    Thanks POJ, that is a so much nicer and cleaned of version of what I had. I guess it's a good lesson that I need to take more type and clean my stuff up. Quick question regarding output. My hope was that the script would step through and take the first city name from file1 + the 1st keyword name from line 1 + the first proxy from line 1 and use that for the first query and then do the same with the second, third, forth line and so on. It looks like it's using the 1st keyword with the 1st city and then going through all of the proxies. 

    Keyword: [Myeloid+leukemia+lab]
Current proxy:1.0.251.108:8080
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_0.html
Wide character in print at slurpiegui17.pl line 214.

Current proxy:1.1.190.162:8080
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_1.html
Wide character in print at slurpiegui17.pl line 214.

Current proxy:1.161.42.95:8088
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_2.html
Wide character in print at slurpiegui17.pl line 214.

    [download]

    I'll take a look at it right now as well and see if I can figure out how to change it for the desired output. Should be much easier with such clean code. Who knows I might even be abel to figure it out. If I succeed I'll comment on this post. Thanks

[reply]
[d/l]

      OK, consider using shift in place of the inner loops

      for my $city (@cities) {
  my $keyword = shift @keywords;
  my $proxy   = shift @proxies; 
  .
  .

      [download]
      poj
[reply]
[d/l]

        Thanks. I understand the concept and see that the shift would and almost have it working. I'm getting an explicit package name error on $proxies. I'm using my $proxies earlier in the script so I"m not sure why I'm getting the error. I'll keep trying. Here's what I have. 

        # create useragent
    my $ua = LWP::UserAgent->new; 
    $ua->agent('Mozilla/8.0');

    # get pages for each city, keyword, proxy 
    
    for my $city (@cities) {
        my $keyword = shift @keywords;
        my $proxy   = shift @proxies;
        
     
        print "City :    [$city]\nKeyword: [$keyword]\n";

          print "Current proxy:$proxies\n";

          # Use this UA/Proxy to fetch something....
          $ua->proxy(['http'], 'http://'.$proxies);     
          my $url = join '',$link1 . $city ."+". $keyword,
                            $link2 . $city ."+". $keyword,
                            $link3;
          my $response = $ua->get($url);
          print "getting $url\n";      
          if ($response->is_success) {
            my $filename = "Bing/${city}_${keyword}.html";
            print "Creating $filename\n";
            open my $fh, ">", $filename 
                 or die("Could not open $filename. $!");
            print $fh $response->decoded_content;  # or whatever
            close $fh;
          } else {
            die $response->status_line;
          }
          print "\n"; 
        }
    close($fh);

        [download]

        These two lines are causing the errors:

            print "Current proxy:$proxies\n";

          # Use this UA/Proxy to fetch something....
          $ua->proxy(['http'], 'http://'.$proxies);

        [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom