in reply to Re: Converting a parallel-serial shell script
in thread Converting a parallel-serial shell script

Yes - I get about 20 files per month, and these need to be processed before they can be loaded into the DB.

And yes, unfortunately, the loading process cannot read from a pipe, as the server process itself wants to read from the file. I want to avoid fancy stuff like trying to make it read from /proc/$$/fd/4, especially as on that machine there is no /proc filesystem.

Replies are listed 'Best First'.
Re^3: Converting a parallel-serial shell script
by BrowserUk (Patriarch) on Sep 18, 2008 at 12:50 UTC

    Updated: This modified version actually terminates. ++Corion

    Needs some error checking, but this should work:

    #! perl -slw use strict; use threads; use Thread::Queue; our $THREADS ||= 4; my $Qraw = new Thread::Queue; my $Qtsv = new Thread::Queue; sub convert{ $_[ 0 ] } sub toTSV { while( my $filename = $Qraw->dequeue ) { my $outFile = $filename . '.tsv'; open my $fhIn, '<', $filename or warn "$filename : $!" and nex +t; open my $fhOut, '>', $outFile or warn "$outFile : $!" and nex +t; while( <$fhIn> ) { my $tsv = convert( $_ ); print $fhOut $tsv; } close $fhOut; close $fhIn; $Qtsv->enqueue( $outFile ); } $Qtsv->enqueue( undef ); } my @threads = map threads->create( \&toTSV ), 1 .. $THREADS; ## Filenames from command line, 4 threads to terminate $Qraw->enqueue( @ARGV, (undef) x $THREADS ); for( 1 .. $THREADS ) { while( my $tsvFile = $Qtsv->dequeue ) { system "echo $tsvFile"; unlink $tsvFile; } } $_->join for @threads;

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.