RE: Re: How do I remove blank lines from text files?

I don't like this for the following reasons:

You are reading the whole file in memory TWICE! Once for the arguments to the foreach and another one when pushing everything into @data. In any case, you want to use while instead of foreach.
You are overwriting the file in place, without creating a backup. What if the machine crashes in the middle of the "print FILE @data"? You have lost your data.

I personally like the technique of writing the results to a new temporary file, and then renaming the original to a backup name and the temporary file to the original. Something like this:

open FILE, "/path/to/file" or die "$!\n";
open OUT, "/tmp/tmpfile.$$" or die "$!\n";
while(<FILE>) {
  next if /^\s*$/;
  print OUT, $_;
}
close FILE; close OUT;
rename("/path/to/file", "/path/to/file.bak")
  or die "Error in rename: $!\n";
rename("/tmp/tmpfile.$$", "/path/to/file")
  or die "Error in rename: $!";
[download]

Side note: using "/tmp/tmpfile.$$" as a temporary file name could have security implications if the program is running set-uid. For better ways of creating a temporary file name, see the FAQ How do I make a temporary file name?

Also, a regex is not necessary if you are looking for strictly empty lines. But many times, a line is considered empty even if it contains white space, in this case using regular expressions is the best way to do it.

--ZZamboni

Comment on RE: Re: How do I remove blank lines from text files? Download Code

Replies are listed 'Best First'.
RE: RE: Re: How do I remove blank lines from text files? by c-era (Curate) on Jul 11, 2000 at 20:57 UTC
? ? ? ? How did you come up with those reasons. It is true that if you open a file for writing and the system crashes, the file is destroyed. But the file is opened in read/write. When the system crashes the file will be unchanged unless the file is closed properly. There may be other reasons to make a backup, you reason is not one of them. As the foreach vs while, on my solaris there is no difference in memory usage between the two. I ran the two programs with a 40MB file and the memory usage was the same. I can understand the file write vs read/write mistake, but next time could you please check you facts before you post.	[reply]
RE: RE: RE: Re: How do I remove blank lines from text files? by ZZamboni (Curate) on Jul 11, 2000 at 22:03 UTC
Wrt the file being opened read/write: if you are rewriting the whole file and it is a large file and the system crashes in the middle, I will argue that the file may be partly rewritten, depending on when the system buffers were last flushed. I don't think the file will remain unchanged until it is properly closed. But I'm willing to be convinced otherwise. Wrt memory usage: my mistake. The immediate problem I saw with using foreach instead of while is that foreach provides an array context, whereas while provides a scalar context. Therefore, when you use foreach, the <FILE> slurps the entire file at once and creates a list for the foreach to cycle through. Then you are pushing each element into @data, from where I assumed the data would be duplicated. However, reading the foreach documentation, I see the following: ...each element of the list is aliased to the loop variable in turn ... Note that the loop variable becomes a reference to the element itself, rather than a copy of the element. Therefore you will not be duplicating data. You are reading the whole file in memory only once, not twice as in my original message. However, I still think reading the whole file in memory is a bad idea, because you will eventually find a file that will not fit in memory. Unless there are other reasons for doing it, I will always prefer to process it one line at a time. --ZZamboni	[reply]