cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

Happy Easter/Passover/Sunday, bros, depending on your persuasion.

I am just getting the hang of this threading stuff, so forgive me if this is a clueless question but...

I am running a script of which this is a simplified example (processresponse is a sub not included):

#!/usr/bin/perl
use  strict;
use LWP::UserAgent;
use Proc::Queue size => 3;
use POSIX ":sys_wait_h"; # imports WNOHANG
my $pid;
my $user_agent = LWP::UserAgent->new;
$user_agent->timeout(30);
my @urlstoget = qw( http://foo.com http://bar.com http://etc.com http:
+//and-so-on.com );
my $starttime = time;
while (@urlstoget) {
        my $url  = shift(@urlstoget) };
        $pid = fork;
        if(defined ($pid) and $pid==0) {
            my $request = HTTP::Request->new('GET', $url);
            my $response = $user_agent->request($request);
            my $dbh = connectdb('listeningpost');
            processresponse($response)
            exit(0);
       }
   1 while waitpid(-1, WNOHANG)>0; # reaps childs
}
my $elapsed = time - $starttime;
open LOG,">getitemshtmlog.txt";
print LOG "$n urls in $elapsed seconds, ",$elapsed/$n," sec per url\n"
+;

[download]
Thing is, everyting in the while loop runs fine, but the stuff after it doesn't execute. Why not?

TIA...

Steve

Replies are listed 'Best First'.
Re: fork/threads: running after childrem
by kyle (Abbot) on Apr 08, 2007 at 22:33 UTC
    1. You have a stray closing brace after shift(@urlstoget).
    2. You use Proc::Queue, but I don't have that, so I commented it out. Maybe that's the problem. It appears to do about what Parallel::ForkManager does, so maybe using that would help.
    3. Global symbol "$n" requires explicit package name
    4. There's no semicolon between the call to processresponse() and exit.

    After I fixed each of those (and commented out the actual fetching and connecting to the db), it did just what I expected. Could you provide a running example that shows the problem?

[reply]
[d/l]
[select]
Re: fork/threads: running after childrem
by salva (Canon) on Apr 09, 2007 at 07:37 UTC
    The problem I can see in your code is that you are not waiting for the last children to exit before writing to the log file. I would write is as follows: 
    #!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;
use Proc::Queue size => 3, qw(run_back waitpids);

my $starttime = time;

my $user_agent = LWP::UserAgent->new;
$user_agent->timeout(30);

my @urlstoget = qw( http://foo.com http://bar.com http://etc.com http:
+//and-so-on.com );

my @pids;
for my $url (@urlstoget) {

    push @pids, run_back {
        my $request = HTTP::Request->new('GET', $url);
        my $response = $user_agent->request($request);
        my $dbh = connectdb('listeningpost');
        processresponse($response)
    };
}

waitpids(@pids);

my $elapsed = time - $starttime;

open LOG, ">", "getitemshtmlog.txt" or die "unable to open log file";

print LOG "$n urls in $elapsed seconds, ",$elapsed/$n," sec per url\n"
+;

    [download]

    If this version doesn't work yet, it could be a bug on Proc::Queue, in that case send the author (=> me) a bug report with your full script or post it here.

[reply]
[d/l]
      Thank you that did it.

      BTW I built the questionable code based on the example code at CPAN . The code you just gave would make a better example, IMO.

[reply]
Re: fork/threads: running after childrem
by dk (Chaplain) on Apr 09, 2007 at 17:30 UTC
    Most probably you'll want POSIX::exit vs just exit.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom