Tried increasing $BUFSIZE ||= 2**31; but got this err Negative length at...

As you probably worked out, the parameter is being used as a signed 32-bit value, and 2**31 rolls over to a negative value.

Tried subtracting 1 in this manner $BUFSIZE ||= 2**31-1 and got this err Out of memory during "large" request for 2147487744 bytes, total sbrk() is 108672 bytes at ./test.pl line 9. I then ran ulimit which came back as 'unlimited'. I'm betting SA's will not change server config or recompile kernel on my behalf so is this a dead-end?

I have no knowledge of Solaris at all, but I think that whilst your server has 16GB of ram, it is probable that each of the 8 cpu's is limited to 2GB. This is a very common upper limit with 32-bit OS's. The theorectic upper limit is 4 GB, but often the other 2GB of each processes virtual address space is reserved by the OS for it's own purposes.

For example, under NT, MS provide a set of APIs collectively known as "Address Windowing Extensions" that allow individual processes to access memory beyond the usual 2GB OS limits by allocating physical ram and mapping parts of it into the 32-bit/4GB address space. But the application needs to be written to use this facility, and it comes with considerable restrictions.

The point is that settling for 2**30 is probably the best you would be able to do without getting a lot more familiar with the internals of your OS.

That said. I would try 2**21, 2**22 & 2**23 first and carefully log the timings to see if using larger buffers actually results in better throughtput. It is quite possible that the extra worked required by the OS in marshalling that volume of contiguous address space will actually reduce your throughput. Indeed, you may find that you get just as good a throughput using 2**16 as you do with 2**20. It may even vary from run to run depending on the loading of the server and a whole host of other factors.

Using ever larger buffers does not imply ever increasing throughput. It's fairly easy to see that if the standard read size is (say) 4k and your processing a 50 GB file, then your going to do 13 million read-search&modify-write cycles and therefore 13 million times any overhead involved in that cycle.

If you increase your buffer size to 2**20, then you reduce that repetiton count to 50 thousand and thereby reduce the effects of any overhead to just 4%. And your OS will have no problems at all in allocating a 1 MB buffer, and reading the 1 MB from disk will easily happen in one timeslice, so there is little to negate the gain.

If you increasing your readsize to 2**22, then your overheads reduce to less than 1% of the original, but only 25% of the 2**20. Worth having, but diminishing returns. Allocating 4 MB will again be no problem, but will the read still complete in a single timeslot? Probably, but you may be pushing the boundary. Ie, it is possible that you will introduce an extra delay through missing a possible timeslot whilst waiting for IO completion.

By the time you get to 2**30, your gain over the 1 MB slot is very small, but you are now forcing the OS to marshall 1 GB of contiguous ram for your process, which itself may take many missed timeslots. And then asking the disk subsystem to read/write 1 GB at a time, which again will definitely introduce several missed timeslots in IOWAIT states. Overall, the gains versus losses will probably result in an overall loss of throughput.

There is no simple calculation that will allow you to determine the breakpoints, nor even estimate them unless the machine is dedicated to the is one task. The best you can do is time several runs at different buffer sizes and look for the trends. In this, looking to maximise your processes cpu load, is probably your best indication of which direction you are heading. My best guess is that you will see little improvement above around 4 MB reads&writes, but the breakpoint may come much earlier, depending upon disk subsystem as much as anything else.

Now we come to multitasking. In all cases, the problem will come down to whether your OS+Perl can correctly manage sharing access to a single file from multiple concurrent threads-of-execution (threads or processes). I'm not familiar with the operation of SysV memory mapping, though I think it may be similar to Win32 File Mapping objects. These would certainly allow processes or threads to process different chunks of a large file concurrently in an efficient and coordinated manner, but the APIs are non-portable and require a pretty deep understanding of the OS in question to use. I don't have that for Solaris so cannot advise, but there are is theSys::MMap module and I noticed that PerlIO has a ':mmap' layer, but it doesn't work for my OS so I am unfamailiar with it.

Now to my threaded code. I have tried two different approaches to this.

My first attempt, tried to overlap the IO and processing by reading into buffers on one thread and doing the translation and writing on a second thread. The idea was that if Perl/C runtime could handle this, I could then try and use more buffers and balance the number of read threads with the number of write threads to get best throughput. On my OS, something is being cached somewhere such that file gets corrupted.

The code I tried is pretty unsophisticated, but was enough to convince me that it wouldn't work:

#### WARNING: THIS CODE IS BROKEN. DO NOT USE ####
#! perl -slw
use strict;
use threads;
use threads::shared;

our $BUFSIZE    ||= 2**20;

open my $fh, '+< :raw', $ARGV[ 0 ] or die $!;

our $buffer1 : shared = '';
our $buffer2 : shared =  '';
our $p1 :shared = 0;
our $p2 :shared = 0;
my $done : shared = 0;
{
    lock $buffer2;
    async {
        while( not $done ) {
            {
                lock $buffer1;
                $p1 = $p2 + length $buffer2;
                sysseek $fh, $p1, 0;
                $done = !sysread $fh, $buffer1, $BUFSIZE;
            }
            {
                lock $buffer2;
                $p2 = $p1 + length $buffer1;
                sysseek $fh, $p2, 0;
                $done = !sysread $fh, $buffer2, $BUFSIZE;
            }
        }
        print "Thread done: $done";
        $done = 1;
    }->detach;
}
sleep 1;

while( not $done ) {
    {
        lock $buffer1;
# A visible, reversable transformation for testing!
        $buffer1 =~ tr[0*][*0]; #
        sysseek $fh, $p1, 0;
        syswrite $fh, $buffer1;
    }
    {
        lock $buffer2;
        $buffer2 =~ tr[0*][*0];
        sysseek $fh, $p2, 0;
        syswrite $fh, $buffer2;
    }
}
close $fh;
[download]

My second attempt--which works (for me)--uses one thread to do all the reading, the main thread to do the transformation, and a third thread to do the writing. Again, the idea is to overlap the IO with the transformation, allowing the process to make best use of the timeslots it is allocated by utilising the time when the read/write threads are blocked in IO wait states to do other stuff. The data is passed between the threads via a couple of queues.

The problem with this is that the iThreads model requires such shared data to be duplicated. It also has to be synchronised. Whilst this works on my system, as I do not have multiple cpus, I cannot tell you whether it will result in greater throughput or not. Nor whether it will continue to work correctly on a multi-cpu machine.

So, I provide it only as an example--testing in your environment is down to you:).

Because of the way iThreads work, and the way this is coded, I would suggest sticking with fairly small buffers 2**20 or 2**22 (and maybe 2**16 would be worth trying also).

I'd appreciate any feedback you can give me if you do try this.

#! perl -slw
use strict;
use threads;
use Thread::Queue;

our $BUFSIZE    ||= 2**20;

open my $fh, '+< :raw', $ARGV[ 0 ] or die $!;
my $size = -s $fh;

my $Qin = new Thread::Queue;
my $Qout= new Thread::Queue;

async {
    my $buffer;
    while( sysread $fh, $buffer, $BUFSIZE ) {
        $Qin->enqueue( $buffer );
        sleep 1 while $Qin->pending > 10;
    }
    $Qin->enqueue( undef );
}->detach;

my $Tout = async{
    sleep 1 until $Qout->pending;
    sysseek $fh, 0, 0;
    while( my $out = $Qout->dequeue ) {
        syswrite $fh, $out;
    }
};

sleep 1 until $Qin->pending;
while( my $in = $Qin->dequeue ) {
    $in =~ tr[0*][*0];
    $Qout->enqueue( $in );
}
$Qout->enqueue( undef );

$Tout->join;

close $fh;
[download]