The idea would probably work also for forked worker processes, but you would need to pass the $pid of the parent process as well as a fileno; since filehandles used by the same owner are writable by all processes of that owner.

The IO::Select loop in the main thread would be similar in setup, to a socket-watch program. As the data comes in to $select->can_read, it will read the data( preferably with sysread in huge chunks), and just copied to an output filehandle.

A few points the OP would have to watch are

1.Making sure the original huge file split dosn't split in the middle of a line, rendering a few records broken.

2. Making sure that IO::Select dosn't clog up and slowdown the output of some threads, by 1 overly aggressive thread outputting too much and hogging the Select object. One possible solution would be to use the largest filehandle buffers possible on the platform, so slower threads can keep outputting to the buffers, if one thread's output becomes very heavy.

The code should be fairly straightforward, and possibly someone as agile with thread code as you, could whip out some code quickly. For me, it would take all morning, and I prefer f'ing off. :-)

you, could whip out some code quickly.

That is my intent, to get something working and see how it fairs. But I'm still not getting a clear picture from your description. A bit confused.

The file handles the main thread is selecting on, are these the same ones that your passed their filenos to threads for duping? Does that mean that the main thread is can-reading on rw dups of teh same files that the threads are writing to?

IF you could knock up a little code to show the flow -- it doesn't have to work, I can knock it into shape -- then it would probably be quicker than me asking 20 questions trying to work out which filehandles do what :)

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

IF you could knock up a little code to show the flow -- it doesn't have to work, I can knock it into shape

Well I ran into some glitches, and the net result is it's probably easiest just to let each thread write to it's own separate file. Of course, your superior insights may see a way out.

I had to create a dummy file, in order to get some fileno's, and although select does seem to intercept the thread writes, they still go to file directly, so there dosn't seem to be any use for the select, except to intercept the data as it's being written to disk. Also the select seemed to repeat it's data reads, but that probably could be fixed.

Before I saw the above glitch, my idea was to have each thread search thru it's list for primes, and only print back to main when a prime was found in it's range.

Conclusion: My original suggestion of letting each thread print to it's own output file, and merging them after script completion, is probably best. Maybe if one used an event loop system, a filehandle watch could be used without the need for a disk file to get a fileno, but then you would be displaying results to a widget of some sort.

#!/usr/bin/perl
use warnings; 
use strict;
use threads;
use IO::Select;
use FileHandle;

my @ranges = ( [1,10000000],[10000001,20000000],[20000001,30000000],
               [30000001,40000000],[40000001,50000000] );

my $sel = new IO::Select();

# thread launching

foreach (@ranges){
  my $fh = FileHandle->new();

  open ($fh,'+>', './dummyfile'); 
# needed to get filehandle to give a fileno
# maybe better to use IO::Handle and give it
# a fileno directly?

    my $start = $_->[0];
    my $end = $_->[1];
    my $fileno = fileno($fh);
    print "$start $end $fileno\n";

  threads->create( \&thread, $start, $end, $fileno )->detach;
  
  $sel->add($fh);
}

# watching thread output
print "Watching\n\n";
#while( scalar (threads->list) > 0 ){  # dosn't seem to work

while(1){     
       
     foreach my $h ($sel->can_read){
         my $buf;
         if ( (sysread($h, $buf, 1024) > 0 ) ){ 
           print "Main says: $buf\n"; 
           #truncate $h, 0;  # bad idea :-)
         }
     }
}


sub thread{

my( $start, $finish, $fileno ) = @_;

open my $fh, ">&=$fileno" or warn $! and die;
print $fh  "thread# ",threads->tid()," -> $start, $finish, $fileno \n"
+  ;

sleep 5;

print $fh  "thread# ",threads->tid()," -> finishing \n"  ;

}
__END__
[download]

---

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh

Hi again, I figured out a way with pipes. It uses IO::Pipe and is a bit tricky, because you need to pass the IO::Pipe object to the thread, before you call writer or reader on it. Otherwise IO::Pipe throws an error about IO::Pipe::End. But this works as proof of concept. It does not have any disk files involved to avoid the fileno problem. Also, since I detached the threads, you need to hit control-c to exit, or work out a method of detecting when all threads are finished, probably thru a shared variable.

#!/usr/bin/perl
use warnings; 
use strict;
use threads;
use IO::Select;
use IO::Pipe;

my @ranges = ( [1,10000000],[10000001,20000000],[20000001,30000000],
               [30000001,40000000],[40000001,50000000] );

my $sel = new IO::Select();

# thread launching

foreach (@ranges){

    my $pipe = IO::Pipe->new();
    my $start = $_->[0];
    my $end = $_->[1];
    print "$start $end $pipe\n";

  threads->create( \&thread, $start, $end, $pipe )->detach;

  # only call reader after pipe has been passed to thread  
  $sel->add( $pipe->reader() );
}

# watching thread output
print "Watching\n\n";

while(1){     
     
     foreach my $h ($sel->can_read){
         my $buf;
         if ( (sysread($h, $buf, 1024) > 0 ) ){ 
           print "Main says: $buf\n"; 
         }
     }
}


sub thread{

my( $start, $finish, $pipe ) = @_;

my $wh = $pipe->writer;
$wh->autoflush(1);

print $wh  "thread# ",threads->tid()," -> $start, $finish, $pipe \n"  
+;

sleep 5;

print $wh  "thread# ",threads->tid()," -> finishing \n"  ;
sleep 2;

}

__END__
[download]

---

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh