I see this sometimes also. It's for sure a bug somewhere. It's a strange one as it's non-deterministic. You can set your program to crawl exactly the same pages and run it several times as a test. The point at which it starts having this bug will vary within a few gets. It won't crash out on the same page each time, but usually within 0-5 gets of one page.

One method that often fixes it for me is to completely quit Firefox (check that no firefox ps's are leftover) and restart it.

Sometimes even that won't fix it though. I find having your program go slower (insert lots of 5-10s sleeps) will help with the problem. In my case my scrapers remember "seen" pages and so once I make it past the problematic area the bug usually disappears again for a while.

In reply to Re: WWW::Mechanize::Firefox Stability Issues when downloading many pdfs by Anonymous Monk
in thread WWW::Mechanize::Firefox Stability Issues when downloading many pdfs by oli_latham

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.