javascript links and robots

dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

Does anyone know if there are any robots out there which can spider through javascript links? Going through the discussion boards the general consensus seems to be "not right now but probably sometime in the future".

If a robot were required to do this what command could be issued?

p.s. I'm aware of the Javascript and Javascript::Toolbox modules, but since the documentation is so sparse (and I've so far received no reply from the author) I can't figure out if they are the answer.

Comment on javascript links and robots

Replies are listed 'Best First'.
Re: javascript links and robots by tilly (Archbishop) on Feb 21, 2004 at 02:25 UTC
It is possible to do this, but it is impossible to do this perfectly. The problem is that JavaScript is a Turing-complete language. Therefore without actually executing all JavaScript that you run into, you can't always tell what it is going to do. That said, you can guess. One guess is that if you see anything that looks like a URL, like http://some.domain.com/stuff, then that is probably going to be reachable somehow. Likewise the same thing URI encoded is also likely to be reachable somehow. With a few heuristics like that, you'll be able to figure out things enough to be useful, but not enough to be really reliable. As you improve the reliability of the heuristics (remember, heuristic is a fancy way of saying "it doesn't work"), the problem rapidly gets harder. If you want more detailed advice, I'd suggest contacting cogent. He faced almost exactly your problem in a past job and had a mostly working solution (albeit he went through nightmares to get there). He has also been working on trying to parse JavaScript well enough to do better than the simple heuristic that I gave above, but I don't think that he is making enough progress. (Nudge him a little...)	[reply]

Replies are listed 'Best First'.

Re: javascript links and robots
by tilly (Archbishop) on Feb 21, 2004 at 02:25 UTC

The problem is that JavaScript is a Turing-complete language. Therefore without actually executing all JavaScript that you run into, you can't always tell what it is going to do.

That said, you can guess. One guess is that if you see anything that looks like a URL, like http://some.domain.com/stuff, then that is probably going to be reachable somehow. Likewise the same thing URI encoded is also likely to be reachable somehow. With a few heuristics like that, you'll be able to figure out things enough to be useful, but not enough to be really reliable. As you improve the reliability of the heuristics (remember, heuristic is a fancy way of saying "it doesn't work"), the problem rapidly gets harder.

If you want more detailed advice, I'd suggest contacting cogent. He faced almost exactly your problem in a past job and had a mostly working solution (albeit he went through nightmares to get there). He has also been working on trying to parse JavaScript well enough to do better than the simple heuristic that I gave above, but I don't think that he is making enough progress. (Nudge him a little...)

[reply]