in reply to Re^6: how to split huge file reading into multiple threads
in thread how to split huge file reading into multiple threads
Well I ran into some glitches, and the net result is it's probably easiest just to let each thread write to it's own separate file. Of course, your superior insights may see a way out.
I had to create a dummy file, in order to get some fileno's, and although select does seem to intercept the thread writes, they still go to file directly, so there dosn't seem to be any use for the select, except to intercept the data as it's being written to disk. Also the select seemed to repeat it's data reads, but that probably could be fixed.
Before I saw the above glitch, my idea was to have each thread search thru it's list for primes, and only print back to main when a prime was found in it's range.
Conclusion: My original suggestion of letting each thread print to it's own output file, and merging them after script completion, is probably best. Maybe if one used an event loop system, a filehandle watch could be used without the need for a disk file to get a fileno, but then you would be displaying results to a widget of some sort.
#!/usr/bin/perl use warnings; use strict; use threads; use IO::Select; use FileHandle; my @ranges = ( [1,10000000],[10000001,20000000],[20000001,30000000], [30000001,40000000],[40000001,50000000] ); my $sel = new IO::Select(); # thread launching foreach (@ranges){ my $fh = FileHandle->new(); open ($fh,'+>', './dummyfile'); # needed to get filehandle to give a fileno # maybe better to use IO::Handle and give it # a fileno directly? my $start = $_->[0]; my $end = $_->[1]; my $fileno = fileno($fh); print "$start $end $fileno\n"; threads->create( \&thread, $start, $end, $fileno )->detach; $sel->add($fh); } # watching thread output print "Watching\n\n"; #while( scalar (threads->list) > 0 ){ # dosn't seem to work while(1){ foreach my $h ($sel->can_read){ my $buf; if ( (sysread($h, $buf, 1024) > 0 ) ){ print "Main says: $buf\n"; #truncate $h, 0; # bad idea :-) } } } sub thread{ my( $start, $finish, $fileno ) = @_; open my $fh, ">&=$fileno" or warn $! and die; print $fh "thread# ",threads->tid()," -> $start, $finish, $fileno \n" + ; sleep 5; print $fh "thread# ",threads->tid()," -> finishing \n" ; } __END__
|
|---|