CPU usage is minimal, it appear to be waiting for more data from the Oracle box. We are working with the administrators to determine what is going on from their side. I believe it is somehow connected to firewall issues.

I've used truss and it shows that the last thing issued was a read. The point where it freezes looks like this: 

time()                                          = 1236374108
write(2, " [ F r i   M a r     6  ".., 70)      = 70
write(8, "02 T\0\006\0\0\0\0\003 ^".., 596)     = 596
read(8, 0x003A17C6, 2064)       (sleeping...)
read(8, "07DB\0\006\0\0\0\0\010\0".., 2064)     = 1380
read(8, "\0\0\0\006 H a r r i s\0".., 631)      = 631
brk(0x003AC710)                                 = 0
brk(0x003B2710)                                 = 0
read(8, "07DB\0\006\0\0\0\0\0\0FF".., 2064)     = 2011
read(8, 0x003A17C6, 2064)       (sleeping...)

[download]

The '(sleeping...)' line is always the last thing printed.

We've ruled out the IPS (Intrusion Prevention System) because we shut it completely off with no apparent change in behavior. We've also snooped the Perl box and it appears to confirm that the program is waiting on read results.

Although I am worried about the cause, I'm more concerned now with finding a methodology that will terminate the program gracefully and alert us to the problem. Even if we do determine the source of the problem (I'm fairly certain we will), we'll still have the threat of similar problems in the future.

In reply to Re^2: Fetching Madness by jlongino
in thread Fetching Madness by jlongino

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.