Hello perlmonks,

I am newbie to Perl & working on the Perl script to perform an action similar to V-Lookup.

So,

As an input I have some large sized text files around 200 MB. These text files are to be searched for all the records present in the another file, say Reference.txt (This file is normally not more than one MB)

I have written script to find all the lines present in these large sized files based on text (string values) in Reference.txt file. All the found records are then written into a new file per each large file iteration.

The script works fine for normal size like 30-40 MB but when it goes more than 100 MB or so. It throws out of memory error.

I have designed these operations as subroutine and calling them.

The code goes something like this...

open (FILE, $ReferenceFilePath) or die "Can't open file";
 chomp (@REFFILELIST = (<FILE>));
 open OUTFILE, ">$OUTPUTFILE" or die $!;
 foreach my $line (@REFFILELIST) 
 {        
 open (LARGEFILE, $LARGESIZEDFILE) or die "Can't open File";
 while (<LARGEFILE>)
 {
     my $Result = index($_, $line);
     if ($Result >  0) 
     {
        open(my $FDH, ">>$OUTPUTFILE");
        print $FDH $_;
     } 
 }
 close(LARGEFILE);
 }
 close(OUTFILE);
close(FILE);

[download]

Can you please guide me on where I am going wrong and what would be the best way to address this issue?

Thanks in advance.

FR

In reply to Out of Memory Error : V-Lookup on Large Sized TEXT File by TheFarsicle

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.