Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Use more than one threads for one file processing

by marioroy (Prior)
on Jun 17, 2016 at 23:58 UTC ( [id://1166013]=note: print w/replies, xml ) Need Help??


in reply to Re: Use more than one threads for one file processing
in thread Use more than one threads for one file processing

Next, workers send data to the manager process via MCE->gather instead of sending to STDOUT.

use strict; use warnings; use MCE::Loop; my @patterns = ( "biopattern1", "biopattern2", "biopattern3" ); my $search = join('|', @patterns); my $regex = qr/$search/; open my $fh, "gunzip -c in.fastq.gz |" or die "open error: $!"; MCE::Loop->init( max_workers => 4, chunk_size => 50, RS => "\n@" ); my @found = MCE::Loop->run( sub { my ( $mce, $chunk_ref, $chunk_id ) = @_; for my $i ( 0 .. $#{ $chunk_ref } ) { if ( $chunk_ref->[$i] =~ /$regex/ ) { MCE->gather( $chunk_ref->[$i] ); } } }, $fh ); MCE::Loop->finish(); close $fh; print join('', @found);

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1166013]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (2)
As of 2024-04-24 15:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found