these pages that the crawler will go to may have 
the link on any of a variety of ways. It may be in a
 frame, it may be generated by the javascript, or
 it may have meta-refresh

[download]
Can I ask you to clarify what you mean by "link"?

It seems from what you wrote above, that what you're actually saying is that the page may contain the URL in any form, not a link as such.

A Meta-Refresh would take you to the other URL, but isn't technically a link. Also, a JavaScript statement like window.location='http://URL.com' would take a JavaScript-enabled browser to the new URL, but again, isn't a link.

Both these examples require that the URL, as a string, be present, but a link requires that it be present and surrounded with the right HTML code.

Apart from recursively testing frames, this shouldn't be hard to do.

--
Weaselling out of things is important. It's what separates us from the animals ... except the weasel.

In reply to Re: Browser Emulation by Cody Pendant
in thread Browser Emulation by jonjacobmoon

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.