Memory corruption from some XS module you've loaded? $socket might be something unusual? Perhaps you could try running perl under gdb? There's nothing about the line you posted that is intrinsicly capable of killing perl.

[Updated: Oh! Perhaps the socket is closed and you're getting SIGPIPE?]

I just added a local $SIG{'PIPE'} = sub { print "got PIPE!\n"; }; handler to the sub with that line. Let's see if that does anything. Since both you and ysth suggest it I'm hopeful that this may indeed be the problem.

Update: Jackpot! It's indeed a SIGPIPE. This also explains why I had trouble creating a smaller sample. There is a timeout on the other side which closes the socket after a certain time of inactivity. As long as at least one of the objects in the queue is making progress there's some communication going and there's no problem. But if all elements in the queue are blocked waiting for a local event the other side will timeout, causing the next attempted write to give the SIGPIPE.

Thanks and ++ to everyone who replied.

Parent process is the bash prompt. Exit code as set in $? is 141.

#!perl
$rc = syswrite $socket, pack("N",$self->{'length'}+length($SP_MAGIC)),
+ 4; 
print "Ok";
[download]

I'm not disregarding the possibility of some bug in perl, but it's likely that the bug originates somewhere else. First up, what's in $self->{length}, $socket and $SP_MAGIC?

Yes I know that won't cause it :)

$self->{length} is 417, $socket is bless( \*Symbol::GEN0, 'IO::Socket::INET' ); according to Data::Dumper, and $SP_MAGIC is a 4-char alfanumeric string.

alright, so does replacing that line with

$rc = syswrite $socket, pack("N",431), 4;
[download]

still produce the error? if it does that would indicate a bug in whatever produced the $socket.
"What should it profit a man, if he should win a flame war, yet lose his cool?"

Sure you can... it's your whole code! :-)

Seriously, now; start removing line by line until it stops failing the way it does (but not failing in a different way; simply running and not doing anything will do the trick).

Not with 100% certainty, but I'm rather sure you'll manage to decrease the quantity of code that produces that behaviour :-)

The problem is that it only happens about 10 minutes into the run.

The main loop of the program is something like this:

while (not_done_yet) {
  if (queue not full) {
    add new element to queue
  }

  @new_queue = ();
  foreach $element (@queue) {
    if (state==1) { blah blah; state=2 }
    ...
    if (timed_out) { next; }
    push $element, @new_queue;
  }
  @queue = @new_queue;
}
[download]

The issue does not show up if I decrease the queue size from the normal 50 to 5, or the timeout from the default 600 seconds to 30. It shows up after the queue has completely filled up (takes a few minutes), and the first few elements have timed out.

I'll see what I can do to decrease it, but I'm not hopeful.

Just a suggestion - first I'd try replacing the variables you're outputing in that line one-by-one, and see if the content of any particular variable (or IO channel) is responsible.

After that, it's just another bug-hunt. The only thing I've learnt about a bug-hunt is to never give up the assumption that it's your code that sucks... ;)

How do you know that that is the line it is dying on?

If you pull the pack out onto a seperate line does the problem persist? Assuming it does, can you replace the packed value with a constant and does the problem persist? If you change the constant does the problem persist? If you replace the pack parameters with constants does the problem persist?

I've added print statements before and after the point where I knew it was dying (by looking at the output), and basically kept adding more closer together until I got to this line. The print before shows up, the print after doesn't.

It did just occur to me that I may not have explicitely disabled buffering on STDOUT, so I'll run it again that way to make sure I'm not just missing some messages that were still in the buffer.

If I replace the packed values with constants then the program won't run. This line gets called successfully hundreds of times already before the program dies, and always with different values in the packet. I can try modifying enough of the callers to pretend I got a successful reply, but that's basically gonna take away all of the socket interaction.

I wouldn't be absolutly sure that that was where the problem was even with buffering turned off, however it is worth a try.

More important is to try and print out the parameters to the pack and see if those are important to the issue. If it fails with specific issues you can then use a debugger and have it break when the appropriate line is reached with the danger parameters.

---

DWIM is Perl's answer to Gödel

#perl, v5.8.8 built for x86_64-linux-thread-multi

_debug( "WRITE to sock (" . length($tmData) . "));
my $bytesWrote = $itosSock->syswrite($tmData);

#perl signals SIGPIPE with a return code 141 (will not execute the nex
+t line)
_debug( "WROTE ($bytesWrote) to sock" );


# I use socat as the client-side and it closes.
[download]

See my response to diatolevi. You have to add a SIGPIPE handler to your code.

Mine is as simple as:

$SIG{PIPE} = sub {
  print STDERR "SIGPIPE @_\n";
};
[download]

Which basically just ignores the SIGPIPE and leaves it to you to handle the error after the syswrite