Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re^2: Adding parallel processing to a working Perl script

by Jim (Curate)
on Apr 21, 2014 at 15:06 UTC ( [id://1083027]=note: print w/replies, xml ) Need Help??


in reply to Re: Adding parallel processing to a working Perl script
in thread Adding parallel processing to a working Perl script

Thank you for your reply, sundialsvc4.

Yes, this is the sort of thing I'm doing now.

start "01" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"A*/Z* 1^> 0 +1.csv 2^> 01.txt start "02" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"B*/Z* 1^> 0 +2.csv 2^> 02.txt start "03" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"C*/Z* 1^> 0 +3.csv 2^> 03.txt start "04" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"D*/Z* 1^> 0 +4.csv 2^> 04.txt start "05" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"E*/Z* 1^> 0 +5.csv 2^> 05.txt start "06" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"F*/Z* 1^> 0 +6.csv 2^> 06.txt start "07" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"G*/Z* 1^> 0 +7.csv 2^> 07.txt start "08" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"H*/Z* 1^> 0 +8.csv 2^> 08.txt start "09" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"I*/Z* 1^> 0 +9.csv 2^> 09.txt start "10" cmd /c perl CountFilesRecords.pl "ABCD/foo bar/"J*/Z* 1^> 1 +0.csv 2^> 10.txt

Blech!

It's precisely what I want to automate using Perl. It seems to me the best way to solve the problem is to add parallel processing of the folders and files inside the counting Perl script itself.

Replies are listed 'Best First'.
Re^3: Adding parallel processing to a working Perl script
by zentara (Archbishop) on Apr 21, 2014 at 16:18 UTC

      Thanks again, zentara.

      Ugh. The whole reason I chose to use Capture::Tiny (after much research) is because it purports to solve all the problems that all the other methods of forking and exec-ing on Windows do not. I read its author's presentation titled How (Not) to Capture Output in Perl.

      I certainly hope I don't have to abandon using Capture::Tiny now. After all, it's working brilliantly. What's failing is something related to my use of Parallel::ForkManager.

        What's failing is something related to my use of Parallel::ForkManager.

        Would you be willing to try a manual fork method?

        For instance:

        #!/usr/bin/perl #by Zaxo of perlmonks my @apps = ( [qw(/path/to/foo args of foo)], [qw(/path/to/bar args of it here)] ); #to avoid Zombies $SIG{CHLD} = 'IGNORE'; for (@apps) { defined(my $cpid = fork) or die $!; $cpid or exec {$_->[0]} @$_ or die $!; } exit 0; # the parent #If the apps don't run as daemons, you may need to have #them ignore SIGHUP or else call &POSIX::setsid so that #the parent's exit doesn't trigger an early demise of the kids.
        or maybe try the following, remembering to try system(1,$cmd) instead of exec( $cmd)
        #!/usr/bin/perl my $child=fork(); # Spawn off a child if ($child>0) { #parent exec ($app1) || die "Could not exec $app1:$!"; exit(0); # Should never get here! } elsif ($child == 0 ) { # Child process exec ($app2) || die "Could not exec $app2:$!"; exit(0); # should never get here either } else { die "Could not fork! $!"; } # Evaluating $! will give you the reason you couldn't # spawn the exec'ed process.

        I'm not really a human, but I play one on earth.
        Old Perl Programmer Haiku ................... flash japh

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1083027]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (3)
As of 2024-04-19 19:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found