in reply to Re^6: WWW::Mechanize::Firefox delayed returns / slow
in thread WWW::Mechanize::Firefox delayed returns / slow

Maybe it is the call to ->content that is slow? Consider using ->selector() and/or ->xpath to extract the element and then ->{innerHTML} to get at its contents.

All your changing around of the ->_wait_while_busy subroutine will only destabilize the whole thing as your script will not wait anymore for Firefox to signal it is ready. Doing this without knowing when and why to do it will only end in tears. It is an internal method and should not be ignored lightly (and if you want to ignore it, there most likely are routines more to the point than this one).

Replies are listed 'Best First'.
Re^8: WWW::Mechanize::Firefox delayed returns / slow
by mascip (Pilgrim) on Apr 09, 2014 at 16:22 UTC

    Thank a lot Corion! For future reference, I will explain how your solution solved my problem (4 years later).

    I am scraping a page which makes ->content() sooo slow. I tried to replace it with 

    my $sel = ->selector('body', single => 1);
my $content = $sel->{innerHTML};

    [download]
    and that second line (a simple assignment) was still as slow. I copied $content to a file, which weighs almost 1MB. It doesn't seem that big. Is 1MB too big for an assignment?

    Anyway, I chose a more restrictive selector and now it's blazing fast. If you come across the same problem, this solution might save you too.

[reply]
[d/l]

      Thank you very much for sharing how you made your script faster!

      That "simple assignment" is not so simple behind the scenes. It involves converting the DOM in the Firefox process to a text string and then copying that text string to the Perl process. The network connection between Firefox and Perl is not as fast as it could be, unfortunately.

[reply]
        I was assuming that ->{innerHTML} was already a string, but you're saying that it's an object, and it's the stringification that takes a long time? But... I didn't ask for it to be stringified yet.
[reply]

In Section Seekers of Perl Wisdom