Re: Use more than one threads for one file processing

Hi heavenfeel,

The following is a demonstration using the MCE::Loop module. What is nice about MCE is the ability to process a file containing multiple records. The record separator for a fastq file is "\n@" which anchors @ at the start of the line. MCE detects "\n" at the start of the record separator.

MCE is a chunking engine allowing a worker to receive several records at a time. The effect is a reduction in the number of trips to and from the MCE-manager process.

The logic below allows one to search multiple patterns. Simply change the patterns to suite your needs. Perhaps, this can read patterns stored in a file. Anyway, this is a small MCE demonstration. The MCE->print statement prints the entire record to STDOUT.

use strict;
use warnings;

use MCE::Loop;

my @patterns = ( "biopattern1", "biopattern2", "biopattern3" );
my $search   = join('|', @patterns);
my $regex    = qr/$search/;

open my $fh, "gunzip -c in.fastq.gz |" or die "open error: $!";

MCE::Loop->init(
    max_workers => 4,
    chunk_size => 50,
    RS => "\n@"
);

MCE::Loop->run( sub {
    my ( $mce, $chunk_ref, $chunk_id ) = @_;

    for my $i ( 0 .. $#{ $chunk_ref } ) {
        if ( $chunk_ref->[$i] =~ /$regex/ ) {
            MCE->print( $chunk_ref->[$i] );
        }
    }

}, $fh );

MCE::Loop->finish();

close $fh;
[download]

Kind regards, Mario

Comment on Re: Use more than one threads for one file processing Download Code

Replies are listed 'Best First'.
Re^2: Use more than one threads for one file processing by marioroy (Prior) on Jun 18, 2016 at 00:08 UTC
The following preserves output order if desired. Here, the chunk_id value is used for ensuring ordered entry into the @found array. It is important for workers to gather even though sending an empty @results. The reason is that the Manager process must know that a given chunk_id has completed. use strict; use warnings; use MCE::Loop; use MCE::Candy; my @patterns = ( "biopattern1", "biopattern2", "biopattern3" ); my $search = join('\|', @patterns); my $regex = qr/$search/; open my $fh, "gunzip -c in.fastq.gz \|" or die "open error: $!"; my @found; MCE::Loop->init( max_workers => 4, chunk_size => 50, RS => "\n@", gather => MCE::Candy::out_iter_array(\@found) ); MCE::Loop->run( sub { my ( $mce, $chunk_ref, $chunk_id ) = @_; my @results; for my $i ( 0 .. $#{ $chunk_ref } ) { if ( $chunk_ref->[$i] =~ /$regex/ ) { push @results, $chunk_ref->[$i]; } } MCE->gather( $chunk_id, @results ); }, $fh ); MCE::Loop->finish(); close $fh; print join('', @found); [download]	[reply] [d/l]
Re^2: Use more than one threads for one file processing by marioroy (Prior) on Jun 17, 2016 at 23:58 UTC
Next, workers send data to the manager process via MCE->gather instead of sending to STDOUT. use strict; use warnings; use MCE::Loop; my @patterns = ( "biopattern1", "biopattern2", "biopattern3" ); my $search = join('\|', @patterns); my $regex = qr/$search/; open my $fh, "gunzip -c in.fastq.gz \|" or die "open error: $!"; MCE::Loop->init( max_workers => 4, chunk_size => 50, RS => "\n@" ); my @found = MCE::Loop->run( sub { my ( $mce, $chunk_ref, $chunk_id ) = @_; for my $i ( 0 .. $#{ $chunk_ref } ) { if ( $chunk_ref->[$i] =~ /$regex/ ) { MCE->gather( $chunk_ref->[$i] ); } } }, $fh ); MCE::Loop->finish(); close $fh; print join('', @found); [download]	[reply] [d/l]


Your skill will accomplish what the force of many cannot
	PerlMonks