in reply to Re: Parallel running the process
in thread Parallel running the process

thanks roboticus. yes definitely this is a way to do it. But This involves lot of manual work to be done. I have to run the script monthly and will be putting it inside cron job.

Is there any way that i can do this in a script itself. Like the process running in background and it writes the output in the desired file as and when it completes.

Replies are listed 'Best First'.
Re^3: Parallel running the process
by roboticus (Chancellor) on Mar 06, 2012 at 20:04 UTC

    ariesb2b:

    It doesn't have to be hard. I was thinking that if that approach worked for you, that you could write a simple script to do the overhead for you. Something like (untested):

    #!/usr/bin/perl
use strict;
use warnings;
use autodie;

my $num_jobs=4;

# Split the work up
open my $IFH, '<', 'data.inp';
my @OFH;
open $OFH[$_-1], '>', "data.$_" for 1 .. $num_jobs;
my $cnt=0;
while (<$IFH>) {
    ++$cnt;
    my $FH = $OFH[$cnt%$num_jobs];
    print $FH, $_;
}
close $OFH[$_-1] for 1 .. $num_jobs;

# Do the work
for my $j (1 .. $num_jobs) {
    `perl orig_do_job --infile=data.$_ --outfile=data.out.$_  & `;
}

# Collect the results
`cat data.out.* >data.out`;

    [download]

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

[reply]
[d/l]
Re^3: Parallel running the process
by JavaFan (Canon) on Mar 06, 2012 at 19:48 UTC
    Sure. You may want to read about fork. Or threads, if you feel more comfortable. Or the classic solution: a select loop. But I assume you're using the DBI to give you database access, and I don't think the DBI has support to give back control to the program while waiting for the results.

    I would probably use fork, but that's because I understand the concept well, and a simple solution seems to be good enough to solve your problem.

[reply]
[d/l]
[select]
      Or, just a batch file / shell script to launch the set of instances in the background.
[reply]

In Section Seekers of Perl Wisdom