in reply to Re^9: Scraping Ajax / JS pop-up
in thread Scraping Ajax / JS pop-up

So a quick update, to help anyone looking for a similar solution.

I have a working scraper bot now, which handles the info in the AJAX/JS pop-up. I've had to resort to sniffing the HTTP with tools/browser plug-ins. I then mimic the HTTP POSTs that went over the wire using HTTP::Request::Common. This was the solution I was trying to avoid (as discussed above in this thread), primarily because if a bot needs to be more autonomous than mine, such as crawling, a more programmatic / self-contained solution is preferred. This is what I was trying to explain to Anonymous Monk. I tried several modules and ways to do that without success. But I should note to those who want to try, that I did not exhaust trying all routes that had potential, so more work with something like WWW::Mechanize::Firefox could possibly be fruitful.

If your scraper is specific to a stable site or does not need to be an autonomous crawler, I would recommend to just cheat the complexity and sniff / mimic the HTTP as described in this thread.

Replies are listed 'Best First'.
Re^11: Scraping Ajax / JS pop-up
by Anonymous Monk on Feb 25, 2012 at 22:43 UTC

    primarily because if a bot needs to be more autonomous than mine, such as crawling, a more programmatic / self-contained solution is preferred. This is what I was trying to explain to Anonymous Monk.

    That was easily understood. Your insistence that it needs to be pure-perl is the problem.

[reply]
      I don't recall "insisting". I do, however, recall seeking, with due diligence (as would, say... a Monk?). ...for a Perl solution on a Perl forum. The reason being, explained above. There is nothing intrinsically limiting about Perl (or most languages with its degree of flexibility) that prevent it from being able to do this. So the expectation is reasonable. As it turns out, this topic continues to be an area active improvement effort in the Perl community.
[reply]

        I don't recall "insisting"...

        Naturally :) I must have imagined this thread going past Re^3

        So the expectation is reasonable

        Actually only hope is reasonable, not expectation.

        Disbelief at the amount of work involved and incredulity that it doesn't just work, on the other hand is unreasonable.

        As it turns out, this topic continues to be an area active improvement effort in the Perl community.

        You can help that effort by submitting a self-contained demonstration of the bug , and better yet, a patch to fix it as well :)

[reply]

In Section Seekers of Perl Wisdom