Hi Monks, I have to filter >20GB files, just pulling any line that has a user specified keyword. It processes tab delimited file and searches through each field for keyword. I have the single threaded version here (please excuse my novice coding/commenting): 
 
use strict;
use warnings;
use Data::Dumper;

#declarations
my @rawfile;
my $filter_count=0;

#input
open(INFILE, "<$ARGV[0]") || die "cannot open file:$!";

####
#filter file
#------------------
#  parse entire file
my $header = <INFILE>;    #grabbing header
chomp $header;
my @headerArray = split("\t", $header);
my $sizeheader=@headerArray;

for my $line (<INFILE>) {
    chomp $line;
    my @splitline = split("\t", $line);
    #my $poskey = $splitline[0] . ":" . $splitline[1];

    for (@splitline){
    if (/^$ARGV[1]/){
        $filter_count++;
        push(@rawfile,$line);
    }
    }
  
}
close INFILE;
print "Completed filtering $ARGV[0]\n";
print "Found $filter_count elements\n";
my $outfilename = substr($ARGV[0],0,length($ARGV[0])-4)."_filter.txt";
print "Filtering to output file: $outfilename\n";

####
#output file
#------------------
open(OUTFILE, ">$outfilename") || die "cannot open file to write: $!";
print OUTFILE "$header\n";

for (@rawfile) {
    print OUTFILE "$_\n";
}
close OUTFILE;

[download]
I hope this will be a good example to learn threading from. Thanks!

In reply to How to split file for threading? by diyaz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.