Re^2: Perl Filehandle?

Thank you for reply, Originally, that is what i had in my mind - work on the files without loading the whole file into memory, butthen i though - given that i need to parse it 1 time for patching, 2 times for deleting duplicates and once for fixing the format, wouldn't i lose time on opening/closing files? I think i even tried to do this once in another script, and opening file alone would take some time. Was i wrong?

Comment on Re^2: Perl Filehandle?

Replies are listed 'Best First'.
Re^3: Perl Filehandle? by wojtyk (Friar) on Nov 15, 2006 at 00:17 UTC
File operations aren't always as white and black as they seem. The operating system does alot of file caching behind the scenes, so you probably won't take as bad a performance hit as you'd think if you open and close the same file several times. Have you tried Tie::File yet? It also does caching and deferred writes and other types of optimization. More importantly, you can give an upper limit on the amount of memory you want Tie::File to consume, which could possibly prevent excessive swapping. However, that discussion aside...my main point (which I think you might have missed) was that I don't think you *have* to parse it 4 times. I could be wrong (as I don't know all the facts), but can't any of this be done in tandem? Ie, why can't the format fix be done at the same time as the patch? You would be patching and formatting lines (not arrays) of data on the fly. You only need a single iteration of all that data, instead of several. You could probably even do the dup checking at the same time. Just build a hash of "things seen" as you're patching/formating, and skip any dups that appear in the hash. Pseudo-code for what I'm talking about: `while( $line = <INFILEHANDLE> ) { chomp($line); $seen{$line} = 1; # for dup checking if ($seen{$line}) { next; } # for dup skipping/deleting patch_line($line); format_line($line); # ... any other code ... print OUTFILEHANDLE $line; }` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Perl Filehandle?
by wojtyk (Friar) on Nov 15, 2006 at 00:17 UTC

Have you tried Tie::File yet?
It also does caching and deferred writes and other types of optimization.
More importantly, you can give an upper limit on the amount of memory you want Tie::File to consume, which could possibly prevent excessive swapping.

However, that discussion aside...my main point (which I think you might have missed) was that I don't think you have to parse it 4 times. I could be wrong (as I don't know all the facts), but can't any of this be done in tandem? Ie, why can't the format fix be done at the same time as the patch?

You would be patching and formatting lines (not arrays) of data on the fly. You only need a single iteration of all that data, instead of several. You could probably even do the dup checking at the same time. Just build a hash of "things seen" as you're patching/formating, and skip any dups that appear in the hash. Pseudo-code for what I'm talking about:

while( $line = <INFILEHANDLE> ) {
  chomp($line);
  $seen{$line} = 1; # for dup checking
  if ($seen{$line}) { next; } # for dup skipping/deleting
  patch_line($line);
  format_line($line);
  # ... any other code ...
  print OUTFILEHANDLE $line;
}
[download]

[reply]
[d/l]