Oftimes, while I find that I can write a more efficient regex than one I find in a program that I am working on, I ask myself two questions:

1. Will it be harder to maintain?
2. How necessary is optimizing the program?

For example in one script, I had to write the following regex:

$input =~ /^(?:[^_]|_(?!${value}))+_${value},((?:[A-Z]\d{1,2},?)+).*/;
[download]

$input =~ /$value,((?:\w\d{1,2},?)+)/;
[download]

Cheers,
Ovid

How does the rant go?

Don't prematurely optimize!
Don't prematurely optimize!
Don't prematurely optimize!

Instead modularize your code and after the fact, if it matters, you are in a position to recognize the big scale issues and redesign. Optimizing your operations won't give you order of magnitude improvements. Fixing your algorithms will.

I think the most important difference between slurping an entire file and reading line-by-line, is that it is pretty easy to have a file so large that slurping it will not just be slower than reading line-by-line, it will fail.

Of somewhat lesser concern is that the line-by-line method scales linearly while the slurp method will start to slow down rather dramatically as the files start to get too large.

So if you have an operation that can be done fairly efficiently in a line-by-line manner, I think you should almost always do it that way.

If you are doing a small file, then the speed-up of slurp mode probably just isn't enough to make much difference. If you are doing large files, then you can't risk the possible huge slow down or out right failure.

If you are doing large files and need as much speed as possible, then you often read and write files chunk-by-chunk (which has the extra advantage of working even when you have a file containing a single line that is too large to fit in your available virtual memory space). This requires the use of read() or sysread() and possibly syswrite(). (A "chunk" is usually a fixed-length and fairly large buffer, like 64K).

But this gets complicated quickly. No simple answer is going to cover all cases.

I tend to process most files line-by-line if they have line-based data. If they have information that can span lines (or records), I slurp up the whole thing. Depending on file size, available memory, and other processes, slurping isn't a good idea.

In this case, line by line seems like it would be more efficient. You can stop reading when you hit the record you want. (Of course, if you'll be doing this sort of thing often, I'd put everything in a database or at least a tied hash, and let something besides Perl handle the searching -- probably a little faster.)

open(FILE, "<file") or die "error opening $!";
my $buf='';
my $leftover='';
while(read FILE, $buf, 4096) {
  $buf = $leftover.$buf;
  my @lines = split(/\n/, $buf);
  $leftover = ($buf !~ /\n$/) ? pop @lines : "";
  foreach (@lines) { 
    # process one line of data
  }
}
close(FILE);
[download]

Note: the open statement should have the filename in the debugging die on failure like it says in perlstyle. Also there are enough levels of buffering that I don't know that worrying about "optimal block size" really makes sense. And finally just letting Perl worry about the line by line is probably faster and more reliable IMO. It will do that buffering behind the scenes for you.

OTOH I have used similar code when working with binary data. So the general technique is good to know.

Good point. Since I also mentioned reading chunks at a time, I'll emphasize that this is not a good idea if you are going to split each chunk into lines.

When you use Perl's <FILE>, Perl itself is reading the file as chunks and splitting them into lines to give to you. I can assure you that you can't do this faster in Perl code than Perl can do it itself. And the Perl code has been tested a lot more than any similar code you might write.

Yes, Tilly already said all of this. I just didn't think he said it strong enough (and I felt guilty for suggesting chunk-by-chunk after not fully understanding a previous reply).

        - tye (but my friends call me "Tye")