in reply to Out of Memory Error : V-Lookup on Large Sized TEXT File

I am not sure about what you are trying to accomplish.
A few data lines and expected output would help quite a bit!
Here is one problem that I see: 
#usr/perl -w
use strict;

my $inPath = "somepath";
my $outPath= "outfile";
my $largefile = "large file";

open (INFILE, "<", $path2File) or 
    die "Can't open $path2File for read";

open (OUTFILE, ">", $outPath) or
     die "Can't open $outpath for write";
     
open (LARGE, ">",$largefile) or
     die "Can't open $largefile for write";
     #I don't see why this is necessay!
     
#do something here.... 
while (<INFILE>)
{  ....
   print OUTFILE "some_data\n";
}
     
#in general, open the files that you
#need to use at the beginning of the program and then
#use those file handles. 
#A "re-open" of a file handle for append is very 
#"expensive" thing within a loop in terms of performance.
#Don't do that.

[download]

Replies are listed 'Best First'.
Re^2: Out of Memory Error : V-Lookup on Large Sized TEXT File
by TheFarsicle (Initiate) on May 03, 2015 at 07:29 UTC

    Thanks Marshall for the reply.

    Actually, The requirement is more like V-Lookup functionality in Excel. Except for two columns, we have two files.

    It means that, There is a file A which is large in size in range of 150-200 MB. This file A contains information about work orders (like Order No, Order Name, Supplier No, Supplier Name, Created Date...and so on)

    There is another file B, which contains only Supplier No for particular region. This file is generally less than 1 MB..around 700 KB something.

    Now, I have to write those records in file C (a new file, kind of output file) for which Supplier No in file B matches with Supplier No in file A.

    So, If you look at the code that I have written, I take the file B contents in a list & then for each Supplier No in file B, I iterate the large file A line by line & check if Supplier No is present in the line. If so, I write the line into file C.

    Can you please suggest now, where I am going wrong?

[reply]
      I take the file B contents in a list & then for each Supplier No in file B, I iterate the large file A line by line & check if Supplier No is present in the line. If so, I write the line into file C.

      You are doing it the wrong way around. You are having to process your entire 200MB fileA, for every line in fileB. That's O(N2).

      Guessing your fileB contains 10-digit Supplier No records, that means your processing will end up reading 70,000 * 200MB ~= 14TeraBytes. (14,000GB). Very slow.

      Now invert your logic. Place the Supplier Nos from fileB into a hash.

      Then read a line from fileA, extract the Supplier No and look to see if it exists in the hash (O(1)), if it does, write a record to fileC.

      This way you read fileB once and fileA once. Just 201MB to read from disk, and ~ 70,000x faster.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
[reply]
      I think that I posted a relevant reply.

      In general you want to read the input file(s) once. That is because this is an "expensive" operation in terms of I/O performance.

      If you wind up with a scenario where for each line of an input file B, you have to re-read each line of input file A, that is very inefficient. And it will take a lot of MIPs (N*N).

[reply]
[reply]

In Section Seekers of Perl Wisdom