Re: Crash with ForkManager on Windows

I suspect your problems are related to the additional $pm->finish and next statements.

Your instantiation of the Parallel::ForkManager object may also be an issue. What you have is different from every example shown in the documentation; also see "perlobj: Invoking Class Methods" (including its "Indirect Object Syntax" subsection).

Using exactly the same message to identify your different cases is pointless: use meaningful messages.

I rewrote your code like this (pm_1199760_parallel_forkmgr_link_checker.pl):

#!/usr/bin/env perl -l

use strict;
use warnings;

use LWP::Simple;
use Parallel::ForkManager;

my @urls = qw{
    http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif
    http://hooboy.no-such-host.int/
    http://www.yahoo.com
    http://www.ora.com/ask_tim/graphics/asktim_header_main.gif
    http://www.guardian.co.uk/
    http://www.pixunlimited.co.uk/siteheaders/Guardian.gif
};

my $max_processes = $ARGV[0] || 8;

my $pm = Parallel::ForkManager::->new($max_processes);

print 'Start link checking.';

for my $url (@urls) {
    $pm->start and next;

    my ($type, undef, $mod) = head($url);

    if (defined $type) {
        print "PASS: $url";
        print '  MOD: ', $mod || 0;
    }
    else {
        print "FAIL: $url";
    }

    $pm->finish;
}

$pm->wait_all_children;

print 'Link checking completed.';
[download]

Here's the output with no argument:

$ pm_1199760_parallel_forkmgr_link_checker.pl
Start link checking.
FAIL: http://hooboy.no-such-host.int/
PASS: http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif
  MOD: 1354074654
PASS: http://www.guardian.co.uk/
  MOD: 0
FAIL: http://www.ora.com/ask_tim/graphics/asktim_header_main.gif
PASS: http://www.yahoo.com
  MOD: 0
FAIL: http://www.pixunlimited.co.uk/siteheaders/Guardian.gif
Link checking completed.
[download]

I then ran it with arguments of 8, 5, 3, 2 and 1. Lke the first run (with no argument), all ran to completion: no crashes! With the exception of 1, they produced the same results (although the order of output varied).

Running with an argument of 1 rather defeats the purpose of running processes in parallel; however, as you said you'd used it, I tried it also. Here's what I got:

$ pm_1199760_parallel_forkmgr_link_checker.pl 1
Start link checking.
PASS: http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif
  MOD: 1354074654
FAIL: http://hooboy.no-such-host.int/
PASS: http://www.yahoo.com
  MOD: 0
FAIL: http://www.ora.com/ask_tim/graphics/asktim_header_main.gif
FAIL: http://www.guardian.co.uk/
FAIL: http://www.pixunlimited.co.uk/siteheaders/Guardian.gif
Link checking completed.
[download]

I ran that a second time: the result was the same. I'll leave you to investigate that further.

— Ken

Comment on Re: Crash with ForkManager on Windows Select or Download Code

Replies are listed 'Best First'.
Re^2: Crash with Parallel::ForkManager on Windows by Discipulus (Canon) on Sep 21, 2017 at 09:27 UTC
Hello kcott, where are you running your code? Unfortunately on Perl (v5.14.2) built for MSWin32-x86-multi-thread on win7 and `Parallel::ForkManager 1.19` I get more than 50% of crashes, using your code. Testing more times mine, as posted above, I get ~25% of crashes.. (not saying mine is better: better is no crash at all ;=) Unfortunately windows is a shaggy beast and I'm not able to debug this behaviour further: what I can add is that during installation I got some warning about disappeared PIDs even if the overall testing process ended ok: t/00-report-prereqs.t ..... ok t/01-utf8-all.t ........... skipped: This is a bug in perl itself on W +indows t/02-callback.t ........... # This test can take 10-20 seconds, please + wait. Started at Thu Sep 21 11:23:08 2017 t/02-callback.t ........... ok t/03-callback-data.t ...... # This test can take 2-6 seconds, please w +ait. Started at Thu Sep 21 11:23:25 2017 t/03-callback-data.t ...... ok t/basic-methods.t ......... ok t/changing-pids.t ......... ok t/start_child.t ........... ok t/waitpid-conflict.t ...... # code outside of P::FM stole -1152 child process '-1152' disappeared. A call to `waitpid` outside of Para +llel::ForkManager might have reaped it. t/waitpid-conflict.t ...... ok t/waitpid-waitonechild.t .. # code outside of P::FM stole -5456 # found warning: child process '-5456' disappeared. A call to `waitpid +` outside of Parallel::ForkManager might have reap ed it. t/waitpid-waitonechild.t .. 1/? # expected to find warning: (?^:child +process '\d+' disappeared. A call to `waitpid` out side of Parallel::ForkManager might have reaped it\.) t/waitpid-waitonechild.t .. ok t/waitpid_blocking.t ...... ok All tests successful. [download] L* PS modified the title a bit for better later search There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re^3: Crash with Parallel::ForkManager on Windows by kcott (Archbishop) on Sep 21, 2017 at 10:03 UTC
G'day Discipulus, Thanks for the follow-up. While it's clear the OP was using Windows, I wasn't aware that this was a Windows-specific problem: I don't see anything like "This worked on XYZ, but not on Windows". I'm using: `Perl 5.26.0 (built for darwin-thread-multi-2level) Parallel::ForkManager 1.19 LWP::Simple 6.26` [download] I had to install Parallel::ForkManager and the `cpan` output is still on one of my screens. Here's my equivalent lines to those you posted: t/00-report-prereqs.t ..... ok t/01-utf8-all.t ........... skipped: Need utf8::all for this test cras +hing on Windows # This test can take 10-20 seconds, please wait. Started at Thu Sep 21 + 17:07:29 2017 t/02-callback.t ........... ok # This test can take 2-6 seconds, please wait. Started at Thu Sep 21 1 +7:07:45 2017 t/03-callback-data.t ...... ok t/basic-methods.t ......... ok t/changing-pids.t ......... ok t/start_child.t ........... ok # code outside of P::FM stole 2369 child process '2369' disappeared. A call to `waitpid` outside of Paral +lel::ForkManager might have reaped it. t/waitpid-conflict.t ...... ok # code outside of P::FM stole 2371 t/waitpid-waitonechild.t .. ok t/waitpid_blocking.t ...... ok All tests successful. [download] You appear to have had problems with `t/waitpid-waitonechild.t` that I didn't have; everything else looks fairly similar. To be honest, I started the installation, went and made a coffee, came back and saw: `YANICK/Parallel-ForkManager-1.19.tar.gz /usr/bin/make install -- OK cpan[2]>` [download] I rarely review the (often many screenfuls of) output unless the installation failed. — Ken	[reply] [d/l] [select]
Re^4: Crash with Parallel::ForkManager on Windows by holli (Abbot) on Sep 21, 2017 at 12:46 UTC
I am getting the bogus tests too, yet I can run this and get no crashes at all. Win 8.1 64-Bit, ActivePerl 5.24 built for MSWin32-x64-multi-thread holli You can lead your users to water, but alas, you cannot drown them.	[reply] [d/l]
Re^5: Crash with Parallel::ForkManager on Windows by kcott (Archbishop) on Sep 22, 2017 at 06:15 UTC
Re^2: Crash with ForkManager on Windows by amitsq (Beadle) on Sep 21, 2017 at 10:18 UTC
Taking your code, i get always the same output regardless which param i take. But it crashes too.	[reply]
Re^3: Crash with ForkManager on Windows by kcott (Archbishop) on Sep 22, 2017 at 06:07 UTC
Feedback, so far, is showing crashes on Windows Perl versions `5.14` and `5.16`; however, `5.24` on Windows did not crash. "CPAN Testers Matrix: Parallel-ForkManager 1.19" is showing failures (for "mswin32") up to `5.22.3`, but no failures for `5.24.0` or higher. There's virtually no failures for any other platforms: two very old "linux" ones for `5.8.x`; and one other for "cygwin" `5.22.2` (which I believe only runs on Windows anyway). You should probably dig a bit deeper into this than I have, but problems do not appear to be solely related to ActivePerl; while looking around I came across this GitHub bug report: "Parallel::Forkmanager 1.19 cpan install fails on Windows Server 2016 Strawberry Perl 5.16.3". (Don't take that one report as being in any way conclusive.) Obviously, installation failures are not the same as crashes, but a trend is suggested. Again, something for you to look into further. I'd suggest installing a newer Perl (at least `5.24.0`; but `5.26.0` — the highest stable version — would be better) and trying the code again with that. Also ensure you've got the latest versions of the modules you're using. — Ken	[reply] [d/l] [select]
Re^4: Crash with ForkManager on Windows -- newer perl does not crash by Discipulus (Canon) on Sep 22, 2017 at 07:25 UTC
Hello kcott and all, I can confirm that Strawberry Perl portable 5.24.1.1 32bit on win7 64bit and Parallel::ForkManager 1.19 run the original code and mine without any crash. Good to know. L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l]