Slight corollary question to my previous scraper node -- although you don't need to read it to answer this one -- I need to extract links from a page, but not all of them.

Now sometimes I can find the links I want on an HTML page just by matching a URL pattern. This method is amenable to parsing with Toke::Parser or similar.

But say a site uses a completely opaque URL format like "?storyid=123456" for everything?

What I've done in the past is to find the chunk of the page which contains those "good" links as a way to exclude the "bad" ones. And I've done it the "dumb" way, i.e.

    $whole_thing =~
      m|<some unique html start string>(.*?)<end string>|s;
    $good_chunk = $1;
[download]

and then working on the $good_chunk.

I've spent a bit of time looking at Toke::Parser and HTML::Parser and I can't seem to figure out how to do the equivalent.

Say I've determined that what I need is

<div id="good_chunk">
[download]

up to the closing tag of that DIV.

I need something like

while ( my $token = $p->get_tag( "div" ) ) {
    if ( $token->[1]->{'id'} eq 'good_chunk' ){
        # get the entire contents of the div, as HTML,
        # for further parsing
    }
}
[download]

Perhaps I'm missing something obvious?