mantyBBQ has asked for the wisdom of the Perl Monks concerning the following question:
Gracious and wise Monks,
I am attempting to thread a simple parsing subroutine using Thread::Pool::Simple. I can add the thread to the thread pool successfully, the threaded sub's args are passed successfully.
From within the thread, I can, of course, print to STDOUT and I am able to call subs that exists in the main script as well as subs that are imported (for example, use DateTime::TimeZone).
I am not able to call anything external from the thread that has not been imported. Before attempting threading, I was able to call the external sub dynamically (while the sub name does not change, each external .pm is specific to a file format).
This used to work (left out some code that us irrelevant):
use strict; use threads; use threads::shared; use Thread::Pool::Simple; use Getopt::Long; # read parameters use Pod::Usage; use Log2t::Common; use DateTime::TimeZone; use Digest::MD5; use Time::HiRes;
then creation of thread pool:
# prepare thread pool my $pool = Thread::Pool::Simple->new( min => 2, # at least 2 workers max => 10, # at most 10 workers load => 3, do => [\&parse_artifact] ) || print "couldn't create thread pool: $!\n";;
then, the input modules (each is for a file format...iehistory.pm is one of these modules) are loaded...BUT...my threads seem to puke when referencing a sub within them
# load all format files eval { # include the format foreach( keys %formats ) { require $format_dir . '/' . $formats{$_}->{'file'}; } };
then recursive directory search and wait for thread pool...within parse_dir, if a file is of an appropriate type, it should be parsed in the thread pool using parse_artifact
# test if we need to parse the artifact and then parse it if ($verify->{'success'}) { $pool->add($formats{$f_key}, $_ ); } </code># now we need to recursively go through every file and directory in th +at we got provided as an input and verify if we can parse it (and the +n parse the file) parse_dir( $dir );
after directory has been recursed...pool waits for completion
# wait for thread pool to empty $pool->join();
here is where things start smelling badly...the parse_artifact() sub gets the args just fine, can print to standard out...and can even call a sub in a loaded module fine...but, when the iehistory.pm is called (mind you, this is just a parser...no OLE or anything odd), the thread just gets stupid...before threading...this ran like the wind
# we have the artifact here, it's verified so we just need to parse it sub parse_artifact() { my ($format, $file) =@_; my $t_line; my @args = undef; print Log2t::Common::get_version(); my $description = Log2t::input::iehistory::get_description(); print "##this was the IE get_description() output: $descriptio +n \n"; # load the log file (prepare, action provided here depends on +the format in question) print "parse_artifact fired; format passed is: $format->{'name +'}; file passed is $file\n";
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: threaded sub cannot access external module
by Corion (Patriarch) on Apr 16, 2010 at 16:22 UTC | |
by mantyBBQ (Initiate) on Apr 19, 2010 at 15:51 UTC | |
by Corion (Patriarch) on Apr 19, 2010 at 16:13 UTC | |
by mantyBBQ (Initiate) on Apr 19, 2010 at 23:28 UTC |