... the problem is only seen under heavy production load

Have you checked the memory consumption (either on a single instance of the script, or summed over multiple instances that might be running simultaneously on a given server)?

If any single instance takes up a significant amount of memory, then you can probably work out how many concurrent jobs it would take to swamp available RAM, and cause one or more of the jobs to go into severe swapping / page faulting. The standard unix/gnu "top" command might suffice to spot a problem of that sort as it's happening.

How hard will it be to reduce the memory footprint of your script? Alternately, how bad will it be to limit the number of simultaneous jobs? The thing about page-fault delays is that the timing impact is non-linear: making one job wait a few sec before it can really start -- to make sure that it is serialized relative to some other job (won't start till some other job finishes) -- can often lead to an overall faster completion than allowing it to run immediately and simultaneously, causing unsustainable competition for available resources,

In reply to Re^3: Debugging running processes by graff
in thread Debugging running processes by kappa

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.