I'd suggest something like:

my @urls = ($url);
my @next_set;

while (@urls) {
    # Storage for links
    my @next_set;
    for my $url (@urls) {
        # Get the page & such
        $mechanize->get($url);
        my $page = $mechanize->content;
        my $title = $mechanize->title;
        print "<b>$title</b><br />";

        # add the pages links to the next list
        push @next_set, $mechanize->links;
    }
    # OK, we've processed all in @urls, so load @urls
    # with all the links we've found since last time,
    # and start over again.
    @urls = @next_set;
}

[download]

I use two arrays here because it's generally not a good idea to modify an array you're iterating on. So we basically use the second array as a bucket to hold all the links we find while processing the first array. Then, when we finish the first array, we reload it and start again.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

In reply to Re: Building a Spidering Application by roboticus
in thread Building a Spidering Application by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.