semio has asked for the wisdom of the Perl Monks concerning the following question:

fellow monks,

I have a simple script that performs a google query, strips excess html and presents the web links, that is everything contained within <a> tags. My previous approach had been to strip off everything surrounding the <a> tags, but I had the thought that simply matching the links would be a cleaner approach. I've been encountering some issues trying to formulate the regex correctly. My regex seems to be matching the entire scalar rather than extracting the result and sending it to my array. Ideally I would like to match everything within <a> tags (including the tags themselves) minus the google ad links and such. This is what I've come up with so far. Any suggestions are greatly appreciated. cheers. 

#!/usr/bin/perl -w

use strict;
use LWP::UserAgent;

my $search = 'perlmonks';

my $ua = LWP::UserAgent->new;

$ua->agent(
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/200
+70309 Fire
fox/2.0.0.3"
);
my $req =
  HTTP::Request->new( GET =>
      "http://www.google.com/search?hl=en&num=100&q=$search&btnG=Googl
+e+Search"
  );
$req->content_type('application/x-www-form-urlencoded');
my $res = $ua->request($req);

my $data;

if ( $res->is_success ) {
    $data = $res->content;
}
else {
    print $res->status_line;
}

my @links = $data =~ m/\<a href(.*)\<\/a\>/gs;

foreach my $link (@links) {
#    if ( $link =~ /google/i ) {
#        next;
#    }
#    else {
        print $link . "this matched\n";
 #   }
}

[download]

Replies are listed 'Best First'.
Re: matching links in a web request
by moritz (Cardinal) on Jun 22, 2007 at 16:29 UTC
    I'd try to match all links, and later filter them.

    For the URLs you could use Regexp::Common::URI, or something like this regex:

    m#\<a href="([^"]+)"\>(.*?)\</a\>#

    Alternatively you could use a module to parse the HTML, maybe that would simplify things.

    Perl 6 in German

[reply]
[d/l]
Re: matching links in a web request
by jhourcle (Prior) on Jun 22, 2007 at 16:43 UTC
    $data =~ m/\<a href(.*)\<\/a\>/gs;

    Regexes are greedy. This pattern will match from the first '<a href' to the last '', which is likely not what you want.

    If you want to continue with your existing logic, you can try:

    $data =~ m/\<a href(.*?)\<\/a\>/gs;

    But, regexes might not be the best for matching this sort of thing, and actually parsing the HTML may be easier to maintain in the long run.

    ...

    However ... in your particular case, Google doesn't like you doing this sort of thing. They used to offer accounts for people who wished to automate searching (with limits on how many searches you could run per day), but they seem to be phasing them out.

[reply]
[d/l]
[select]
      But, regexes might not be the best for matching this sort of thing, and actually parsing the HTML may be easier to maintain in the long run.

      Yep, there's also been a recent CUFP which if nothing else has the merit of showing how to do that easily with HTML::SimpleLinkExtor.

[reply]

Back to Seekers of Perl Wisdom