in reply to WWW::Mechanize::Firefox latency using RemoteObject

You can at least avoid some of the loops by using ->xpath queries to find relevant elements. For example to find all INPUT elements for a given form, you can use

my @inputs = $mech->selector( 'input,select,textarea', node => $form )
+;

[download]

There is no convenient way to do bulk-fetching for attributes, so you're basically stuck there.

If all you want is the HTML of elements, you can just print $element->{innerHTML}

Replies are listed 'Best First'.
Re^2: WWW::Mechanize::Firefox latency using RemoteObject
by hansendc (Novice) on Apr 03, 2011 at 20:14 UTC
    OK, that ->selector() syntax looks interesting.

    Do you see what I'm saying about how long each of the transactions to the browser and back takes, though? I'm curious if you have any thoughts on what's causing that bit. As I said, it seems a wee bit odd that the system is idle instead of cpu or I/O bound.

[reply]

      This is just as you suspected - each attribute access involves at least one roundtrip over TCP from Perl to Firefox and back. There is little you can do except avoid accessing Javascript data from Perl, or to make bulk requests.

      Likely, the CPU time gets split up between Firefox and Perl and the kernel for the Network, and I'm not sure how your monitoring accounts for time spent in the kernel.

      Update: If you start to optimize your application and try to reduce the accesses to Javascript objects, the object bridge has some counters that might be of help:

              stats => {
            roundtrip => 0,  # total number of roundtrips
            fetch => 0,      # number of attribute fetches
            store => 0,      # number of attribute stores
            callback => 0,   # number of callbacks triggered
        },

      [download]
      use Data::Dumper;
my $repl = $mech->repl;
warn Dumper $repl->{stats};

      [download]
[reply]
[d/l]
[select]
        Thanks for the stats tip. That will help a bunch.

        So, I did just put my code in a loop. It sits basically pounding on the roundtrip and fetch operations in the stats. Even when this is happening, the CPU is never more than about 25% utilized, total, including perl, firefox and the kernel.

        I'm measuring using vmstat, which accounts for system time nicely. I also did some kernel profiling to make sure nothing in there looked funny. Nothing really sticks out there.

[reply]

In Section Seekers of Perl Wisdom