I have created a quick experiment based on your question. Not sure what it is worth but I tried to use the 'chunk_size' facility in MCE to achieve what you want

Since you didn't show any data to test on I had to make some assumptions on how it looks like and created a script to generate some data files

The program starts up a Perl data reader program using open3 and feeds the results to the MCE 'mce_loop_f'. The chunk size eventually determines how many lines are being output in a new file. Files are written to a 'output' subfolder

I say again, just is just a funny experiment of mine and maybe someone else can say something about performance, but hey, it might do the job!

test.pl

use strict ;
use warnings ;
use IPC::Open3 qw( open3 ) ;
use MCE::Loop ;

MCE::Loop::init {
   max_workers => 3, chunk_size => 10
} ;

my $cmd = 'perl readfiles.pl' ;

sub do_work {
    my @ar = @{$_[0]} ;
    my $id = $_[1] ;
    open( my $fho, ">", "output\\$id.txt" ) or die "Can't open > outpu
+t\\$id.txt: $!" ;
    foreach ( @ar ) {
        print $fho $_ ;
    }
    close($fho) ;
}

my $pid = open3(
    undef,
    my $filedata,
    undef,
    $cmd, @_,
) ;
print STDERR "Starting MCE loop\n" ;
mce_loop_f {
   my ( undef, $chunk_ref, $chunk_id ) = @_ ;
   do_work( $chunk_ref, $chunk_id ) ;
} $filedata ;
print STDERR "Ended\n" ;

waitpid( $pid, 0 ) ;
warn "Failed\n" if ( $? << 8 != 0 ) ;
close($filedata) ;

[download]

readfiles.pl

use strict ;
use warnings ;

my @files = qw( data1.txt data2.txt data3.txt ) ;
foreach ( @files ) {
    open( my $fh, "<", $_ ) or die "Can't open < $_: $!" ;
    while(<$fh>) {
        chomp;
        print $_ . "\n" ;
    }
    close( $fh ) ;
}

[download]

createdatafiles.pl

use strict ;
use warnings ;

open( my $fh, ">", "data1.txt" ) or die "Can't open > data1.txt: $!" ;
for my $i ( 1..1000 ) {
    print $fh "$i\n" ;
}
close $fh ;

open( $fh, ">", "data2.txt" ) or die "Can't open > data2.txt: $!" ;
for my $i ( 1001..2000 ) {
    print $fh "$i\n" ;
}
close $fh ;

open( $fh, ">", "data3.txt" ) or die "Can't open > data3.txt: $!" ;
for my $i ( 2001..3000 ) {
    print $fh "$i\n" ;
}
close $fh ;

[download]

In reply to Re: Merge and split files based on number of lines by Veltro
in thread Merge and split files based on number of lines by Sekhar Reddy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.