these pages that the crawler will go to may have 
the link on any of a variety of ways. It may be in a
 frame, it may be generated by the javascript, or
 it may have meta-refresh
[download]

Can I ask you to clarify what you mean by "link"?

It seems from what you wrote above, that what you're actually saying is that the page may contain the URL in any form, not a link as such.

A Meta-Refresh would take you to the other URL, but isn't technically a link. Also, a JavaScript statement like window.location='http://URL.com' would take a JavaScript-enabled browser to the new URL, but again, isn't a link.

Both these examples require that the URL, as a string, be present, but a link requires that it be present and surrounded with the right HTML code.

Apart from recursively testing frames, this shouldn't be hard to do.

--
Weaselling out of things is important. It's what separates us from the animals ... except the weasel.