Now if you have a race condition between the query of $q->pending and $q->dequeue (another threads empties the queue in between) you can get a block.

Try $q->dequeue_nb instead.

(Disclaimer: I have no experience with threads, so don't be surprised if that's not the problem...)

I've managed to reproduce this on a single cpu windows machine with AS5.8.8 (using ping instead of smtp) and it appears to be a bug in the handling of backticks within threads.

The symptoms I am seeing are that one thread hangs, never returning from the backticks, even though I can see that all instances of the executable (ping.exe) have terminated. Using this simplified version of your thread code with added trace:

sub SNMPthread  {
    my $thread = shift;
    while ( my $left = $q->pending ) {
        warn "$thread: $left";
        my $ip = $q->dequeue;
        warn "$thread: $left, $ip";
        my $return = `ping $ip`;
        warn sprintf "$thread: $left got %d bytes of output\n", length
+ $return;
        my @ele = split ' ', $return;
        warn "$thread: $left, $ip, $ele[ -4 ], $ele[ -1 ]";
    }
    warn "thread $thread finished\n";
}
[download]

The trace for the hanging thread is:

8: 94 at C:\test\662828.pl line 30.
8: 94,  212.58.227.137 at C:\test\662828.pl line 32.
8: 94 got 471 bytes of output
8: 94,  212.58.227.137, 204ms,, 192ms at C:\test\662828.pl line 36.

8: 88 at C:\test\662828.pl line 30.
8: 88, 30.207.168.74 at C:\test\662828.pl line 32.
8: 88 got 228 bytes of output
8: 88, 30.207.168.74, =, loss), at C:\test\662828.pl line 36.

8: 78 at C:\test\662828.pl line 30.
8: 78, 202.229.106.211 at C:\test\662828.pl line 32.
[download]

As you can see, each IP is producing 4 lines of trace. Two before the backticks and two after. On it's third attempt, the backticks hang. However, the spawned ping instance goes away, so it's internal to Perl rather than the executable where the hang is occuring. The latter also suggested by the fact I can reproduce it using a different executable under a different OS.

I've tried substituting a piped open for backticks and varied the number of threads with the same result. Always one thread only that hangs. So it appears to be an internal problem that is the culprit.

However, I do have a fix for you. Upgrade to 5.10.0. The same code completes successfully using it, which also tends to indicate that it's a Perl problem, that's been fixed already.

dude, nice one!

I wonder if what you've found tho' is perhaps a different issue, as what i have observed is subtly different to what you see.

what i was seeing was a different number of threads that would hang each time i ran the script. if i ran the script with 10 threads i would get about 4-7 threads that would hang.

thanks for your proposed solution and your effort in testing this out in multiple environments! unfort, in a production env with multiple 100's of thousands of customers (and this really being a one off deal) i wont have the opportunity to upgrade to 5.10.0. it is something to keep in mind for the future tho', and perhaps i can initiate a longer term initiative.

if i ran the script with 10 threads i would get about 4-7 threads that would hang.

Quite possibly symptoms of the same problem on a multi-cpu system rather than my single cpu system?

You could try addressing the issue that moritz raised, by changing your loop to:

while( my $left = $q->dequeue_nb ) {
[download]

And see if that helps any. If not, try generating a standalone demo of the problem (using ping) and post a perlbug report. Maybe whomever there fixed the problem for 5.10 will recognise the symptom and be able to offer a patch against 5.8.8?

Beyond that, I can't see any way of avoiding the bug, as it also affects piped opens. The alternative might be to use NET::SNMP to do whatever the executable your using is doing. Perhaps too painful to reproduce?

One thing that route does have going for it is a non-blocking mode that appears to work quite well, which mean that you could avoid using threads altogether.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

 my $retval = `/usr/bin/snmpget -v1 -c <commstring> $ip sysDescr.0`;
[download]

Alarms don't work well in threads, but you could try running a parallel thread to act as a timer, and run the snmpget command thru a piped-open or IPC::Open3 so you get a pid that can be killed when the timer times out.

actually i guess snmpget has an implicit timeout as if i cant connect to a device, i get a timeout message on STDERR :-)

Then run the backtick with `command 2>&1` and it shouldn't hang, as eventually it will get the stderr return.

---

I'm not really a human, but I play one on earth. Cogito ergo sum a bum