Another thing that might help is abstracting the code from each engine-specific section into a subroutine. This way you can debug that code, compartmentalize it, and have a more readable flow of execution in your main program. Also, if you ever want to reuse the code that you wrote to get and parse the website results, moving that code into subroutines is the first step.

Here's some example code for you, it abstracts the Altavista search method into it's own subroutine and also uses Parallel::ForkManager to speed things up:

#!/usr/bin/perl

use LWP::Simple;
use Parallel::ForkManager;
use strict;

$|=1;

my @urls = qw(
   http://sulfericacid.perlmonk.org
   http://sulfericacid.com
);

my $number_of_forks = scalar @urls;
my $forkmanager = Parallel::ForkManager->new( $number_of_forks );

foreach my $site ( @urls ) {
   $forkmanager->start and next;
   my $altavista_results = &altavista_search( $site );
   print "Searched http://www.altavista.com for site $site\n";
   print "results: $altavista_results\n";
   $forkmanager->finish;
}

$forkmanager->wait_all_children;


#######################
# Altavista!
#######################

sub altavista_search {
   my $url = shift;
   my $engine_link = 
      "http://www.altavista.com/web/results?q=link:$url&kl=XX&search=S
+earch";

   my $content = get("$engine_link");
   my @lines = split /\n/, $content;
   
   my $results;
   
   foreach my $line (@lines) {
      $results = $1 if $line =~ m/AltaVista found (.*) results/; 
   }
   return $results;
}
[download]

Some possibly mutually exclusive suggestions.

• You're not parsing HTML at all. That's good, because that would just slow you down.
• Try with $| off. I don't expect it'll make a difference either way. Notice that $| = 1 won't get the data out of Google any faster, so maybe it doesn't do quite what you think.
• When you find the matching line, last out of the loop.
• Don't split your big string at all. Just run the regexp against it.
• Use something more sophisticated than LWP::Simple that will hand you back the stream coming in from the server. Parse the incoming stream and drop it when you find the magic line. Or roll your own HTTP request with IO::Socket::INET.
• Parallelization is your friend, but you can't be faster than the slowest engine. Get benchmarks and attack the slow ones first.

Not an answer, but keep in mind that what you're doing is technically in violation of Google's Terms of Service.

What I am doing really isn't against their Terms of Service in the least, I'm just not doing it the right way yet. It will not let me parse them until I setup the DPI for it, but that's another story. I'm more concerned on how to increase the speed so I can begin adding the other ~8+ engines to the list.


"Age is nothing more than an inaccurate number bestowed upon us at birth as just another means for others to judge and classify us"

sulfericacid

No Automated Querying
You may not send automated queries of any sort to Google's system without express permission in advance from Google.

You are sending automated queries to Google, by the letter of their TOS that's forbidden. You may get away with it, you may get your IP (or worse your ISP's netblock) blocked by Google. Simply pointing out that Google technically forbids any non-browser access not through their official API. Go right ahead and live on the edge (as long as you're not on my netblock :).

use LWP::Parallel;
use HTTP::Request;

my %Engines = )
  google => { url=>'qqq', parse => sub {},},
  av     => { url=>'zzz', parse => sub {},},
)

my $pua = LWP::Parallel::UserAgent->new();
$pua->register($_->{url}, $_->{parse}) for values %Engines;
$pua->wait();
# at this point you have all pages fetched and filtered thru parse sub
+s
[download]