in reply to Re^5: Can't call method "proxy" on an undefined value at
in thread Can't call method "proxy" on an undefined value at

All of the help you have given is greatly appreciated. I'm just a newbie trying to do the best I can. One last question: So I'm trying to put it all together and read three different files and query through the proxy with the values pulled. Basically something like this: 1)use proxy one <value1 for file> 2)to search URL containing keyword<value 1 from 2nd file> + city<value 3 from 3rd file>

I have the for and the while that grabs the keyword and city from their respective files and it work perfectly but I can't seem to figure out how to put the proxy piece in. Nesting a for in a for doesn't seem to work and with the current code I'm not getting all three values changing as expected. Just the proxy value. Here's what I have 

open(my $fh, '<', 'cities.txt') or die $!;
    print $fh;
    open(my $prx, '<', 'proxies.txt') or die $!;
    print $prx;
    
    mkdir 'Bing', 0755;
    mkdir 'Bing/1Parsed/', 0755;
    mkdir 'Bing/1Parsed/Html/', 0755;
    
    chomp(my @cities = <$fh>);
    close($fh);
    
    chomp(my @prx = <$prx>);
    print $prx;
    close($prx);

    open($fh, '<', 'keywords.txt') or die $!;

    for my $city (@cities) {
        seek($fh, 0, 0);

        while (my $keywords = <$fh>) {
            chomp($keywords);
            print "$city $keywords\n";
            
            #toggle proxy selection
    
            my $ua = LWP::UserAgent->new; 
            $ua->agent('Mozilla/8.0');

            for my $p (@prx){
                print "Current proxy:$p\n";
                $ua->proxy(['http'], "http://$p");
                 # Use this UA/Proxy to fetch something....
                 my $xml1 = $link1 . $city ."+". $keywords . $link2 . 
+$city ."+". $keywords . $link3;
                 #my $xmla = $link3 . $row . ".com";
                 #my $xmlx = $link4 . $row;

                 my $filename1 = "Bing/".($city)."_". ($keywords) . ".
+html";
                 open my $fh1, ">", $filename1 or die("Could not open 
+file. $!");
                 
                 
                my $xml2 = get $xml1;
                 print $xml1;
                 print "\n";
                 print $fh1 $xml2;

    
    
    
    }
}
}

[download]

When it's late I'm often not as clear headed ass I'd like to be as evidenced by my posts tonight!

Replies are listed 'Best First'.
Re^7: Can't call method "proxy" on an undefined value at
by poj (Abbot) on Feb 09, 2016 at 12:20 UTC

    You might find it cleared using the same approach for all your loops and print statement to check what is happening. Note I have added a number to the output filename to create a different file for each proxy. 

    #!perl
use strict;
use LWP::UserAgent;
use File::Path 'make_path';

my $link1='';
my $link2='';
my $link3='';

# make directories
my $dir = 'Bing/1Parsed/Html/';
unless (-d $dir){
  make_path($dir, { verbose=>1, mode=>0755 });
}

# keywords
open my $fh, '<', 'keywords.txt' or die $!;
chomp(my @keywords = <$fh>);
close $fh;

# cities
open my $fh, '<', 'cities.txt' or die $!;
chomp(my @cities = <$fh>);
close $fh;

# proxies
open my $fh, '<', 'proxies.txt' or die $!;
chomp(my @proxies = <$fh>);
close $fh;
  
# create useragent
my $ua = LWP::UserAgent->new; 
$ua->agent('Mozilla/8.0');

# get pages for each city, keyword, proxy 
for my $city (@cities) {
  for my $keyword (@keywords){
    print "City :    [$city]\nKeyword: [$keyword]\n";

    for my $i (0..$#proxies){
      print "Current proxy:$proxies[$i]\n";

      # Use this UA/Proxy to fetch something....
      $ua->proxy(['http'], 'http://'.$proxies[$i]);     
      my $url = join '',$link1 . $city ."+". $keyword,
                        $link2 . $city ."+". $keyword,
                        $link3;
      my $response = $ua->get($url);
      print "getting $url\n";      
      if ($response->is_success) {
        my $filename = "Bing/${city}_${keyword}_${i}.html";
        print "Creating $filename\n";
        open my $fh, ">", $filename 
             or die("Could not open $filename. $!");
        print $fh $response->decoded_content;  # or whatever
        close $fh;
      } else {
        die $response->status_line;
      }
      print "\n"; 
    }

  }
}

    [download]
    poj
[reply]
[d/l]

      Thanks POJ, that is a so much nicer and cleaned of version of what I had. I guess it's a good lesson that I need to take more type and clean my stuff up. Quick question regarding output. My hope was that the script would step through and take the first city name from file1 + the 1st keyword name from line 1 + the first proxy from line 1 and use that for the first query and then do the same with the second, third, forth line and so on. It looks like it's using the 1st keyword with the 1st city and then going through all of the proxies. 

      Keyword: [Myeloid+leukemia+lab]
Current proxy:1.0.251.108:8080
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_0.html
Wide character in print at slurpiegui17.pl line 214.

Current proxy:1.1.190.162:8080
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_1.html
Wide character in print at slurpiegui17.pl line 214.

Current proxy:1.161.42.95:8088
getting https://www.bing.com/search?q=San+Jose+CA+Myeloid+leukemia+lab
+&go=Submit&qs=n&form=QBLH&pq=San+Jose+CA+Myeloid+leukemia+lab&sc=0-25
+&sp=-1&sk=&cvid=ACB1AECE74E94CA8A48805667D1178F7
Creating Bing/San+Jose+CA_Myeloid+leukemia+lab_2.html
Wide character in print at slurpiegui17.pl line 214.

      [download]

      I'll take a look at it right now as well and see if I can figure out how to change it for the desired output. Should be much easier with such clean code. Who knows I might even be abel to figure it out. If I succeed I'll comment on this post. Thanks

[reply]
[d/l]

        OK, consider using shift in place of the inner loops

        for my $city (@cities) {
  my $keyword = shift @keywords;
  my $proxy   = shift @proxies; 
  .
  .

        [download]
        poj
[reply]
[d/l]

In Section Seekers of Perl Wisdom