The failures really look like as if Firefox fails to load local files. This is very weird.

The Perl program is likely waiting for Firefox/Javascript to send the event that the page has loaded completely. Why this is slow for you, I don't know. What version of Firefox are you running?

I'm not sure what is causing the delay for you. I think that maybe in upcoming versions, I'll add more debugging features that output more diagnostics.

I solved the local file problem. It was indeed just a matter of the tests running as root in cpan cmd line while my firefox was running as non-root. I gave my non-root user access to the files and now the tests are running much better.

Here are the only failures:

t/50-app-new.t .................... 6/8 
#   Failed test 'The name is "Standard"'
#   at t/50-app-new.t line 46.
#          got: 'Default'
#     expected: 'Standard'
# Looks like you failed 1 test of 8.
t/50-app-new.t .................... Dubious, test returned 1 (wstat 25
+6, 0x100)
Failed 1/8 subtests 

t/50-mech-bufsize.t ............... 4/8
[download]

It hangs on the last one at 4/8. I waited 15 mins, but it's hung. The hang seems very similar to the hangs my program is having. I added some debugs to that test and tried again. It's hanging forever on the mech->content_as_png call. Commenting out that call I can get the tests to continue.

t/70-urlbar.t ..................... 1/2 
#   Failed test 'We changed the location once'
#   at t/70-urlbar.t line 48.
#          got: '0'
#     expected: '1'
Use of uninitialized value $this in pattern match (m//) at t/70-urlbar
+.t line 49, <DATA> line 1.

#   Failed test '... to that other page'
#   at t/70-urlbar.t line 49.
#                   undef
#     doesn't match '(?-xism:/70-urlbar-2.html$)'
# Looks like you failed 2 tests of 2.
t/70-urlbar.t ..................... Dubious, test returned 2 (wstat 51
+2, 0x200)
Failed 2/2 subtests
[download]

Also, I get a lot (dozen+) of these warnings:

Subroutine MozRepl::__load_plugins redefined at /usr/lib/perl5/site_pe
+rl/5.10.0/Module/Pluggable/Fast.pm line 104, <DATA> line 1.
[download]

And I had to turn off my proxy settings ("direct connection") to make a few tests pass (my squid seems to return things other than what the tests expect).

t/50-app-new.t .................... 6/8 
#   Failed test 'The name is "Standard"'
#   at t/50-app-new.t line 46.
#          got: 'Default'
#     expected: 'Standard'
# Looks like you failed 1 test of 8.
[download]

Ouch - the name of the "Default" theme in Firefox is localized. I have to take care of that in my tests, likely by not testing for the name but the UUID of the theme.

t/50-mech-bufsize.t ............... 4/8
[download]

If that test hangs, this is bad for large webpages. It loads an image-heavy deviantart.com page, so that should work in any case. Maybe this is an issue with your proxy setup, but without knowing more about it, that'll be hard to replicate.

It would be nice if you could tell me what tests fail (and how) with your proxy, as that would help me make the tests (and maybe the module) more resilient against the proxy results.

t/70-urlbar.t ..................... 1/2 
#   Failed test 'We changed the location once'
#   at t/70-urlbar.t line 48.
#          got: '0'
#     expected: '1'
Use of uninitialized value $this in pattern match (m//) at t/70-urlbar
+.t line 49, <DATA> line 1.

#   Failed test '... to that other page'
#   at t/70-urlbar.t line 49.
#                   undef
#     doesn't match '(?-xism:/70-urlbar-2.html$)'
# Looks like you failed 2 tests of 2.
t/70-urlbar.t ..................... Dubious, test returned 2 (wstat 51
+2, 0x200)
Failed 2/2 subtests
[download]

These test failures are weird, as they also seem to indicate that the script does not receive some Firefox events. I'm not sure what to make of that.

Hi! My firefox is 3.5.15, the latest rpm for Fedora 12. I will be switching to F14 in the not-too-distant future and I'm sure that will update me to 3.6. Maybe that will help.

What local files would MozRepl be trying to load? My firefox is already open and it works fine/normal while browsing hundreds of pages all day long. Hmmm, perhaps because I'm doing the cpan install as root but the firefox is running as my non-root user? I'll investigate that.

I think your guess about waiting is correct. Like I said, things always load immediately when a mech call is made, but the calls then sit there for a long time before returning.

Further tests on simple sites indicate that it probably is just this over-the-top js site I am trying to mech that is causing the problems. The simple sites seem to mech just fine.

Perhaps a "max_wait" option could be added to force mech call returns after X seconds even if they think they still need to wait. Something like "return with what you have at the moment" shortcut.

I'm pretty good with perl and I could certainly help with coding and testing if I get some hints on where to look. The module and its zillion dependencies taken together are quite a daunting pile for me at the moment.

Success! I debugged and hacked some of the pm code and got the mech to go without hangs or overly long pauses.

What I did is just a kludge, but it could hint at the real solution.

I found that the get() was calling synchronize(). Synchronize would be slow (5-20 secs) on the $callback->(). The big problem though is after get()ing a couple of pages the subsequent _wait_while_busy($load_lock) hangs forever.

So in _wait_while_busy I added before the while loop: my $start=time; and added at the end of the for loop: return $element if time-$start>5;

Now my program seems to work as expected! Hooray!

Obviously we're waiting on events that never happen. Again, some global timeout might help. Or, a rejig of this code if this is a true bug (and not just a quirk of the site I'm crawling).

I can continue to help as required. Thanks!

->synchronize waits on the appropriate event to fire in Firefox. If it takes longer than you expect, most likely you're waiting for the wrong event. See the ->events method and the events => argument to the constructor on how to define your own events.

Just adding a timeout in ->_wait_while_busy only papers over the fact that you're not receiving (or rather, listening to) the right event.

Now, finding the right event to listen to requires some ingenuity, as the default events (DOMFrameContentLoaded, DOMContentLoaded, error, abort, stop) don't seem to fire "soon enough" in your case. Maybe load is another good event to listen to, but it fires before subframes have loaded.