Re^2: How does the while works in case of Filehandle when reading a gigantic file in Perl

/me nods ...

A very compelling explanation for “exponentially slower” would be that this program is, indeed, reading the entire file into memory and trying to process it that way. The process’s working set tries to become larger than the file itself, and the memory-manager of the system can&rquo;t accommodate that, and what happens is that the system starts “thrashing.” If you graph the performance-curve of that, it is “exponential.”

If a file is indeed being read linearly, without stuffing it all into memory, then the completion time profile ought to be linear: the “lines processed per millisecond” should be more or less constant, and the working-set size of the process (as seen by top or somesuch) should not vary according to the size of the file being processed. If the file is twice as big, it should take about twice as long, all other things being equal. So, if that is not what is seen to be happening, “a great big slurp” is almost certain to be the explanation, and the good news is that it should be quite easy to fix.

Replies are listed 'Best First'.
Re^3: How does the while works in case of Filehandle when reading a gigantic file in Perl by Anonymous Monk on Jan 30, 2015 at 15:22 UTC
"a great big slurp" is almost certain to be the explanation raj4489 said "I use while for reading it line by line".	[reply]
Re^4: How does the while works in case of Filehandle when reading a gigantic file in Perl by CountZero (Bishop) on Jan 30, 2015 at 20:48 UTC
Actually sundialsvc4 is close to the truth (but not quite there yet). The program repeatedly opens the output file for appending a new line to it. That means that the OS must somehow find the end of the file and whereas that perhaps does not necessarily mean the OS has to "slurp" the whole file, it will have to walk the chain of disk-space to find where is the end of the file, then read the last few sectors in its buffer, append the new line and write out the buffer to disk again (either immediately or upon closing the file). And all this is done for each line added since the program opens and closes the output file for each new line. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James My blog: Imperial Deltronics	[reply]
Re^5: How does the while works in case of Filehandle when reading a gigantic file in Perl by locked_user sundialsvc4 (Abbot) on Jan 30, 2015 at 22:14 UTC
Indeed you are correct, Count ... I completely failed to notice that. A boner like that can definitely torch any program’s performance. Upvotes coming your way (twice) for a “good catch.”