I think moving to requireing the module tree in the thread subs is probably compounding the problems.

Imagine one thread gets a timeslice, gets part way through loading the module tree and then gets interrupted and another thread starts its attempt to load those same modules. If one of them has global state that is only used during loading, that interruption may leave it in an undefined state and the second thread inherits that state at the C level.

If you really want to go that route, you should consider wrapping the require in a critical section to ensure that no two threads can be attempting to load the module tree concurrently.

Hello vr,

One can try loading IO::Handle before spawning workers. That alone is helpful for increasing reliability for modules that involve IO::*. I was able to reproduce threads failing but not after adding the IO::Handle line. Even tested with 100 threads without a delay between them.

use warnings;
use feature 'say';
use threads;
use Thread::Queue;

use IO::Handle;  # <-- important

my $q = Thread::Queue-> new;

my @gang = map async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require Compress::Zlib;
        say threads-> tid;
    }
}), 1 .. 8;

$q-> enqueue( $_ ) for 1 .. 8;
$q-> end;

$_-> join for @gang;
[download]

Kind regards, Mario

Hi again,

Here is the same thing using MCE::Hobo. Similar code, but processes instead.

use warnings;
use feature 'say';
use MCE::Hobo;
use MCE::Shared;

# use IO::Handle;  # <-- loaded automatically by MCE and MCE::Shared::
+Server

my $q = MCE::Shared-> queue;

my @gang = map mce_async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require Compress::Zlib;
        say MCE::Hobo-> tid;
    }
}), 1 .. 100;

$q-> enqueue( $_ ) for 1 .. 100;
$q-> end;

$_-> join for @gang;
[download]

For modules not multi-process safe, another thing one can do on Unix platforms is having MCE::Hobo default to posix_exit to avoid all END and destructor processing.

use warnings;
use feature 'say';
use MCE::Hobo;
use MCE::Shared;

# use IO::Handle;  # <-- loaded automatically by MCE before spawning

MCE::Hobo->init( posix_exit => 1 );

my $q = MCE::Shared-> queue;

my @gang = map mce_async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require Compress::Zlib;
        say MCE::Hobo-> tid;
    }
}), 1 .. 8;

$q-> enqueue( $_ ) for 1 .. 8;
$q-> end;

$_-> join for @gang;
[download]

Note that the posix_exit option is not recommended if constructing an object inside the worker involving a temp file. In that case one may want the worker to exit normally. Anyway, the posix_exit option is there if needed as a last resort.

Taken from the MCE::Hobo manual: Set posix_exit to avoid all END and destructor processing. Constructing MCE::Hobo inside a thread implies 1 or if present CGI, FCGI, Coro, Curses, Gearman::Util, Gearman::XS, LWP::UserAgent, Mojo::IOLoop, Prima, STFL, Tk, Wx, or Win32::GUI.

A lot of modules are not multi-process safe and the reason for setting to 1 automatically. Btw, Prima is now multi-process safe recently.

Kind regards, Mario

So, the problem is not about Compress::Zlib (or poor Text::PDF::Filter... did we torture it beyond convalescing? oops, sorry). The minimal script is then:

use strict;
use warnings;
use feature 'say';

use threads;
use Thread::Queue;

my $q = Thread::Queue-> new;

my @gang = map async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require IO::Handle;
        say threads-> tid;
    }
}), 1 .. 8;

$q-> enqueue( $_ ) for 1 .. @gang;
$q-> end;

$_-> join for @gang;
[download]

which sometimes (often, both Windows and Linux) fails randomly. Further, replacing the require line with

use XSLoader; XSLoader::load 'IO';
[download]

sometimes there are strange warnings, like:

Constant subroutine SEEK_END redefined at threads.pl line 4294967295
[download]

Maybe this line number ("broken source"?) is important, maybe not. Until (if ever) the issue is fixed, maybe it would be good idea to advise (documentation, i.e.) to use IO:Handle in any threaded code? Won't hurt...

As to issue of "multi-process safe modules", I can't get same problem if above script is simply re-written using fork (Linux), perhaps it's unrelated issue?

And, certainly, thanks for showing that even a throw-away, few dozen lines script is as easily written using MCE, as "vanilla Perl", except it "just works", because author has put his knowledge and effort to take care of possible bugs. Thanks, Mario :)