Hi all,

The following script is for my crawler to go to a page and get a list of products of that page and the next page. it is a recursive function but there are two problems with it.
1. I am using HTML:TokeParser and a while loop that gets all the links and it has to check two things class of the link or the href but when I put them both together the program only checks for the class and it skips the href:

    next unless defined($token->[1]{href});
    next unless defined($token->[1]{class});
    next unless $token->[1]{href} =~ /\?page=/ || $token->[1]{class} =
+~ /top10_link/;
    my $urls = $token->[1]{href};
[download]

2. the problem is that it loops forever even after I put all the accepted links in a hash table and I check the hash table before adding the next link & recursing. some how it just goes ahead and adds the existing link to the hash anyway:

if(!$parent{$urls}){
      my $count = keys %parent;
      $parent{$urls} = $i;
      my $count2 = keys %parent;
      print "$i: count1 is: $count and count2 is $count2\n$urls\n";
      if ($urls =~ /page=/){
        print "!!!!!!!!!!!!recursing!!!!!!!!!!!!!\n";
        #print "\n\"$i $title\"\n  $parent{$urls}\n";
        &passing($urls);
      }
    }
[download]

and here is the whole of the script:

#!/usr/bin/perl -w
use strict;
use HTML::TokeParser;
use LWP::Simple;
use URI::URL;

my %parent;


sub passing{

my $url = shift;


my $data = get($url) or die $!;

#the magical parser.
my $p = HTML::TokeParser->new(\$data);
 my $i=0;
while (my $token = $p->get_tag("a")) {
    next unless defined($token->[1]{href});
    next unless defined($token->[1]{class});
    next unless $token->[1]{href} =~ /\?page=/ || $token->[1]{class} =
+~ /top10_link/;
    my $urls = $token->[1]{href};
    $urls =~ s/&PHPSESSID=.*//g;
    $urls = &canonical($urls, "http://www.ash-distribution.co.uk/index
+.php");
    my $title = $p->get_trimmed_text;         
    if (!$parent{$urls}){
      $parent{$urls} = $i;
      if ($urls =~ /page=/){
        print "!!!!!!!!!!!!recursing!!!!!!!!!!!!!\n";
        #print "\n\"$i $title\"\n  $parent{$urls}\n";
        &passing($urls);
      }
    }
    $i++;
}
}
sub canonical{
  if (not $_[0] =~ m%^http://%){
    $_[0] = url($_[0])->abs($_[1]);
  }
  return $_[0];
}

&passing("http://www.ash-distribution.co.uk/index.php?c=%20Ink&sc=Epso
+n%20Replacement%20Cartridges");
[download]

Any help is appriciated, Thanks in advance.

++Update:I figured why the hash dont work if i start the $i with 1 instead of 0 it will work, i dont know why but if i give the value of the hash 0 it doesnt like it but if I give anything else works fine.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.