Yes - I get about 20 files per month, and these need to be processed before they can be loaded into the DB.

And yes, unfortunately, the loading process cannot read from a pipe, as the server process itself wants to read from the file. I want to avoid fancy stuff like trying to make it read from /proc/$$/fd/4, especially as on that machine there is no /proc filesystem.

Updated: This modified version actually terminates. ++Corion

Needs some error checking, but this should work:

#! perl -slw
use strict;
use threads;
use Thread::Queue;

our $THREADS ||= 4;

my $Qraw = new Thread::Queue;
my $Qtsv = new Thread::Queue;

sub convert{ $_[ 0 ] }

sub toTSV {
    while( my $filename = $Qraw->dequeue ) {
        my $outFile = $filename . '.tsv';
        open my $fhIn, '<', $filename or warn "$filename : $!" and nex
+t;
        open my $fhOut, '>', $outFile or warn "$outFile : $!"  and nex
+t;
        while( <$fhIn> ) {
             my $tsv = convert( $_ );
             print $fhOut $tsv;
        }
        close $fhOut;
        close $fhIn;
        $Qtsv->enqueue( $outFile );
    }
    $Qtsv->enqueue( undef );
}

my @threads = map threads->create( \&toTSV ), 1 .. $THREADS;

## Filenames from command line, 4 threads to terminate
$Qraw->enqueue( @ARGV, (undef) x $THREADS );

for( 1 .. $THREADS ) {
    while( my $tsvFile = $Qtsv->dequeue ) {
        system "echo $tsvFile";
        unlink $tsvFile;
    }
}

$_->join for @threads;
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."