Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

perl dying without a trace

by Crackers2 (Parson)
on Oct 24, 2006 at 21:58 UTC ( #580411=perlquestion: print w/replies, xml ) Need Help??

Crackers2 has asked for the wisdom of the Perl Monks concerning the following question:

I'm running into an issue that has me stumped.

I have a program that will reliably die (as in the process exits, not a Perl die at a certain line. There's nothing printed on STDOUT or STDERR. The exact line it's dying on is this:

$rc = syswrite $socket, pack("N",$self->{'length'}+length($SP_MAGIC)), + 4;

I've tried putting an eval around it but that doesn't make a difference either.

To make things worse, I can't find a limited code sample that still displays this behaviour. I tried doing an strace on the process but of course that also makes the problem go away. The full program is single-threaded but includes plenty of socket traffic and system calls.

Any suggestions on how I could proceed to find out where the issue is?

Replies are listed 'Best First'.
Re: perl dying without a trace
by diotalevi (Canon) on Oct 24, 2006 at 22:15 UTC

    Memory corruption from some XS module you've loaded? $socket might be something unusual? Perhaps you could try running perl under gdb? There's nothing about the line you posted that is intrinsicly capable of killing perl.

    [Updated: Oh! Perhaps the socket is closed and you're getting SIGPIPE?]

    ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

      I just added a local $SIG{'PIPE'} = sub { print "got PIPE!\n"; }; handler to the sub with that line. Let's see if that does anything. Since both you and ysth suggest it I'm hopeful that this may indeed be the problem.

      Update: Jackpot! It's indeed a SIGPIPE. This also explains why I had trouble creating a smaller sample. There is a timeout on the other side which closes the socket after a certain time of inactivity. As long as at least one of the objects in the queue is making progress there's some communication going and there's no problem. But if all elements in the queue are blocked waiting for a local event the other side will timeout, causing the next attempted write to give the SIGPIPE.

      Thanks and ++ to everyone who replied.

Re: perl dying without a trace
by ysth (Canon) on Oct 24, 2006 at 22:20 UTC
    An uncaught signal? What does the perl process's parent get as the process status?

      Parent process is the bash prompt. Exit code as set in $? is 141.

Re: perl dying without a trace
by Joost (Canon) on Oct 24, 2006 at 22:23 UTC

      Yes I know that won't cause it :)

      $self->{length} is 417, $socket is bless( \*Symbol::GEN0, 'IO::Socket::INET' ); according to Data::Dumper, and $SP_MAGIC is a 4-char alfanumeric string.

Re: perl dying without a trace
by cog (Parson) on Oct 24, 2006 at 22:23 UTC
    I can't find a limited code sample that still displays this behaviour.

    Sure you can... it's your whole code! :-)

    Seriously, now; start removing line by line until it stops failing the way it does (but not failing in a different way; simply running and not doing anything will do the trick).

    Not with 100% certainty, but I'm rather sure you'll manage to decrease the quantity of code that produces that behaviour :-)

      The problem is that it only happens about 10 minutes into the run.

      The main loop of the program is something like this:

      while (not_done_yet) { if (queue not full) { add new element to queue } @new_queue = (); foreach $element (@queue) { if (state==1) { blah blah; state=2 } ... if (timed_out) { next; } push $element, @new_queue; } @queue = @new_queue; }

      The issue does not show up if I decrease the queue size from the normal 50 to 5, or the timeout from the default 600 seconds to 30. It shows up after the queue has completely filled up (takes a few minutes), and the first few elements have timed out.

      I'll see what I can do to decrease it, but I'm not hopeful.

Re: perl dying without a trace
by Melly (Hermit) on Oct 24, 2006 at 22:40 UTC

    Just a suggestion - first I'd try replacing the variables you're outputing in that line one-by-one, and see if the content of any particular variable (or IO channel) is responsible.

    After that, it's just another bug-hunt. The only thing I've learnt about a bug-hunt is to never give up the assumption that it's your code that sucks... ;)

    Tom Melly,
Re: perl dying without a trace
by GrandFather (Saint) on Oct 24, 2006 at 22:47 UTC

    How do you know that that is the line it is dying on?

    If you pull the pack out onto a seperate line does the problem persist? Assuming it does, can you replace the packed value with a constant and does the problem persist? If you change the constant does the problem persist? If you replace the pack parameters with constants does the problem persist?

    DWIM is Perl's answer to Gödel

      I've added print statements before and after the point where I knew it was dying (by looking at the output), and basically kept adding more closer together until I got to this line. The print before shows up, the print after doesn't.

      It did just occur to me that I may not have explicitely disabled buffering on STDOUT, so I'll run it again that way to make sure I'm not just missing some messages that were still in the buffer.

      If I replace the packed values with constants then the program won't run. This line gets called successfully hundreds of times already before the program dies, and always with different values in the packet. I can try modifying enough of the callers to pretend I got a successful reply, but that's basically gonna take away all of the socket interaction.

        I wouldn't be absolutly sure that that was where the problem was even with buffering turned off, however it is worth a try.

        More important is to try and print out the parameters to the pack and see if those are important to the issue. If it fails with specific issues you can then use a debugger and have it break when the appropriate line is reached with the danger parameters.

        DWIM is Perl's answer to Gödel
Re: perl dying without a trace
by Anonymous Monk on Oct 30, 2013 at 17:15 UTC
    If it means anything. I have the same exact same issue:
    #perl, v5.8.8 built for x86_64-linux-thread-multi _debug( "WRITE to sock (" . length($tmData) . ")); my $bytesWrote = $itosSock->syswrite($tmData); #perl signals SIGPIPE with a return code 141 (will not execute the nex +t line) _debug( "WROTE ($bytesWrote) to sock" ); # I use socat as the client-side and it closes.
    Looking for a work around or fix.

      See my response to diatolevi. You have to add a SIGPIPE handler to your code.

      Mine is as simple as:

      $SIG{PIPE} = sub { print STDERR "SIGPIPE @_\n"; };

      Which basically just ignores the SIGPIPE and leaves it to you to handle the error after the syswrite

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://580411]
Approved by blue_cowdawg
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (2)
As of 2022-10-02 10:07 GMT
Find Nodes?
    Voting Booth?
    My preferred way to holiday/vacation is:

    Results (8 votes). Check out past polls.