{
    # NOTE: code is untested
    my @chunks = ();
    local $/ = \102400;
    while(<$fh>) {
        my $chunk   = $_;
        my $last_rs = rindex($chunk, $/)
        push @chunks, substr($chunk, 0, $last_rs);

        seek($fh, 1, -(length($chunk) - $last_rs));
    }
}
[download]

_________
broquaint

I don't think this will help that much. There is buffering going on under the surface with stdio anyway and it should pick a good blocksize based on the device it is reading from.

The optimimum blocksize is returned by stat as Zaxo pointed out in a recent post. You could use an approach similar to this one (although the seeks are a really bad idea) and choose a multiple of that blocksize if your lines are usually bigger than it is but you would incur the overhead of breaking it up into lines. You are probably better off leaving that to the lower level routines.

-sauoq
"My two cents aren't worth a dime.";

while(read $fh, my $buffer, 128*2**10) {
    $buffer .= <$fh>; # since last line probably crosses buffer border
    for(split /\n/, $buffer) {
        # ...
    }
}
[download]

# we assume 10 fields per record again
while(read $fh, my $buffer, 128*2**10) {
    $buffer .= <$fh>;

    # we dump all records' fields in a big pile, in which
    # every 9th element contains the last field of one record,
    # plus a newline, plus the first field of the next record
    my @in_field_heap = split /\|/, $buffer;

    while(@in_field_heap) {
        # pull the two glued fields apart
        $in_field_heap[9] =~ /^([^\n]*)\n(.*)/;

        # pull out fields of current record incl the glued one,
        # and reinject the second half of the double field
        my @field = splice @in_field_heap, 0, 10, $2;

        # replace the glued double field by its first half
        @field[9] = $1;

        # ...
    }
}
[download]

        # ...
        push @out_field_heap, @field, "\n";

    }
    # single join over the whole batch
    my $out_buffer = join "|", @out_field_heap;


    # but that means we surrounded the newlines with pipes,
    # so fix em
    $out_buffer =~ s/\|\n\|/\n/g;

    print OUTPUT $out_buffer;
}
[download]

Obviously, optimization for speed can decrease your code's legibility and maintainability fast. Be wary of whether you really need it.

Disclaimer: I benchmarked none of these. The read $fh, $buffer, $len; $buffer .= <$fh>; is known to be the fastest block slurping approach however.

If that output acceleration idea works, it might well be applicable to the split acceleration as well.

# 10 fields..
while(read $fh, my $buffer, 128*2**10) {
   $buffer .= <$fh>;
   $buffer =~ s/\n/\|/g;
   my @in_field_heap = split /\|/, $buffer;

   while(my @field = splice @in_field_heap, 0, 10) {
       # ...
   }
}
[download]

Makeshifts last the longest.

If you're running this off of Win32, you can save noticeable time by periodically defragging your drives.

Regardless of the OS, you can save substantial time if the OUTPUT file you're writing is on a different physical drive than the datafiles you're reading. I takes a lot of time (relativly speaking) to move disk heads across the disk to do "read a bit here, write a bit there" operations. If you can rig things so that drive heads move relatively small amounts (e.g., from track to track) while reading or writing, you can win big.

If you have to run everything off of one drive, then consider buffering your writes to OUTPUT. Perl's buffering will wait until a disk block is full before writing, but you can increase the effective buffer size by doing something like the following in your loop.

  push @buffer, join("|", @fields) . "\n";
  $buffer .= "\n";
  if ( --$fuse == 0 ) {
      print OUTPUT @buffer;
      @buffer = ();
      $fuse = $LINES_TO_BUFFER;
  }
[download]

I first thought about this idea when I was thinking about John_M._Dlugosz's related question a couple of days ago. I made some attempts to verify the benefits of it, but ran into another of the 'features' of the OS on my box. This time, that copy file/b con refuses to do anything if the file is over 32k! Not in and of itself a bad idea if you've ever subjected your office colleges to half an hour of random morse code by typing copy *.exe con by mistake on a machine that doesn't have a volume control, but providing no way to override this is unforgivable.

Anyway, the idea. On most OS's, the system utilities should be pretty well tuned for handling fileio buffering - choosing read sizes etc. So rather than complicating your Perl scripts but re-inventing the wheel on buffering over and over, why not let the OS utilities take care of it for you? Something like:

...
local $/=\nn; # big chunks, small chunks whatever
open FH, '< copy bigbinary con |' or die $!;
while(<FH>) {
    binmode(FH);
    # do whatever.
}
[download]

In this way, the system utilities knowledge of appropriate buffer sizes etc. to handle the io efficiently. Additionally, if the action inherently requires large amounts of memory, then it is return to the os when the child process terminates.

There may also be some performance benefits from having the pipe further buffer the data, especially if the Perl program needs to process the input in small chunks.

In John_M._Dlugosz's case, he could use (forgive my not knowing the correct syntax) something like open FH, "grep -bcs -f '$delimiter' <bigbinary |" or die $!; to find the offsets of his records and then use seek on the file to go get his data?

This would be especially useful if the OS in question does something sensible with filesharing for processes requesting read-only access. He could hold his big file open readonly, whilst spawning seperate processes to do the searching.

Rather than reply to just one of the two replies that suggest doing your own buffering in Perl, I'll reply to that concept here.

Perl does some unusual buffer management that can actually make Perl's I/O faster than I/O written in C using <stdio.h>. Unfortunately, this buffer management gets into the guts of stdio.h and so can only be done if your stdio.h data structures are pretty "normal".

The command "perl -V:d_stdstdio" is supposed to report "d_stdstdio='undef'" if Configure couldn't verify that it was safe for Perl do these optimizations and so Perl's I/O is not "fast".

If Perl's I/O is "fast", then the underlying C code is already doing the "read a big buffer and split it up" and doing it more than twice as fast as you could do it yourself in your Perl script. But if Perl's I/O isn't "fast", then you might be able to get nearly a two-fold speed-up by buffering and splitting yourself.

I'd previously been told that Linux fails this verification and so Perl's I/O is more than twice as slow as it would be otherwise (nearly four times a slow?). Checking v5.6.1 on Linux, I find d_stdstdio='define', so I'm not sure if Perl has improved on this or something else is going on.

I'd also heard that Win32 doesn't have fast Perl I/O but it also reports d_stdstdio='define'. So perhaps that test isn't very useful. /:

So it certainly may be worth trying the buffer-and-split in Perl and see if that makes your I/O faster or slower. It is rather sad that Perl w/o "faster" I/O doesn't manage to do buffer-and-split in C as fast as can be done in a Perl script. I think that case was considered "rare" and so didn't get much attention.

Unfortunately, you probably won't squeeze much out of the code you showed us.

If there is a real opportunity for optimization, it is likely to in the code that you represent with your comment:

# Do something nifty with the fields...

-sauoq
"My two cents aren't worth a dime.";

my ($file, $line, @fields);

foreach $file (@files) {
  open(FILE, $file) or die "Nya, nya: $!\n";
  while($line = <FILE>) {
     @fields = split('|', $line); 
     print OUTPUT join("|", @fields);
  } 
}
[download]

This will save you some fractions of a microsecond :)

$, = "|"; 
print @{[1,2,3]};

instead of

print join(...);
[download]