#!/usr/local/perl5.6.1/bin/perl -slw
use strict;
our $BUFSIZE ||= 2**30;
open my $fhi, '+<', "/data/p_dm200/ndm_ip_pull/test_customer1" or die $!;

while( sysread $fhi, $_, $BUFSIZE ) {
tr^M ;
sysseek $fhi, -length(), 1;
syswrite $fhi, $_, $BUFSIZE;
}
close $fhi;

which was tested against an 8,595,447,728 byte file. The time output was:
real 10m5.95s
user 1m48.55s
sys 0m17.24s

An amazing 10 minutes. I checked the output and it looks exactly as expected. I even retested 3 times and each time the results were similar.

Ok, now I am getting greedy and curious as to if this can be optimized more?? I ran top during this session and saw that SIZE and RES were both around 1026M throughout the duration and only 1 cpu seemed used. Would increasing BUFSIZE help performance linearly? If I was capable (and I am not) would either shared memory threads or parallel forks produce big gains? Any other low-hanging fruit?

Perlfan, the ROMIO seemed interesting but I could not find a perl sample. Still it seemed interesting. Anonymous Monk, please forgive my ignorance but what does HD mean?

A sincere thanks to all,
--Paul

Using a larger buffer size may increase throughput slightly, but then it may not. It will depend upon many factors mostly to do with your file system buffering, disk subsystems etc. The easy answer, given it's only taking 10 minutes is to try it.

As far as using threads to distribute the load across your processors is concerned, it certainly could be done, and could in theory, give you near linear reductions per extra processor.

But, and it's big 'but', how good is your runtime library's handling of multi-threaded IO to a single file?

On my system, even using sys* IO calls and careful locking to ensure that only one thread can seek&read or seek&write at a time, something, somewhere is getting confused and the file is corrupted. I suspect that even after a syswrite completes, the data is not yet fully flushed to disk before the next seek&write cycle starts.

So, maybe you could get it to work on your system, but I haven't succeeded on mine, and I am not yet entirely sure whether the problem lies within Perl, the OS, or some combination of the two.

If you feel like trying this, please come back and report your findings. If you need a starting point, /msg me and I can let you have my failing code.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco.

Rule 1 has a caveat! -- Who broke the cabal?

By HD anonymonk means Hard Disk. The seek times to move the heads around a hard drive are slow compared to memory access and geological compared to processor cache. What this means is processes that are dedicated to doing something to a file are normal disk IO bound. Lets not mention network latencies for now.

If you do manage to split this into threads you may actually reduce performance as each time a different thread gets a shot at it, it forces the HD to drag it's heads over to a completely different part of disk. A single thread reading the file sequentially will not be making the heads seek so much, assuming the file is not desperately fragmented on the media.

Then there are other users competing for those heads and tasking them off to the boondocks of the drive as far as your data is concerned which is why it was suggested you kick the lusers to try and get the disk all to yourself.

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

If you do manage to split this into threads you may actualy reduce performance as each time a different thread gets a shot at it it forces the HD to drag it's heads over to a completely different part of disk. A single thread reading the file sequentialy will not be making the heads seak so much, assuming the file is not desperately fragmented on the media.

That may be true if the file is stored on a single disk. But somehow I doubt an 8-way box dealing with 45-50Gb files uses filesystems that are layed out over single disks. It's far more likely some kind of volume manager (either in software using Solstice Disksuite or Veritas Volumemanager, to name two common products used with Solaris, or in hardware, either by using a RAID card, or by having the RAID done by the backend storage, which in turn, could be done by a specific diskarray, or by the network itself (NAS)). Without knowing more about the volume layout and implementation, it's hard to say how much performance is helped by using separate threads or processes. It becomes unlikely that performance will actually decrease, although bad volume setups happen all the time. Often unknowingly, but also because people want to know the disk a certain file is stored on.

Okay. I attempted the above suggestion:

1) Tried increasing $BUFSIZE ||= 2**31; but got this err Negative length at ./test.pl line 9. Tried subtracting 1 in this manner $BUFSIZE ||= 2**31-1 and got this err Out of memory during "large" request for 2147487744 bytes, total sbrk() is 108672 bytes at ./test.pl line 9. I then ran ulimit which came back as 'unlimited'. I'm betting SA's will not change server config or recompile kernel on my behalf so is this a dead-end?

2) BrowserUK, I would like to test threads. If your system was not Solaris 8, please let me know how I /msg you to get your test code. Or post with a caveat.

The last question I had was about leveraging additional cpu's. Can we coax perl to coax the OS to throw some additional CPU's onto the fire? Would this make any difference? Based on the time output above, is it fair to say that this process is completely IO bound? Meaning that adding cpu's would only increase IO WAIT.

Upon searching perlmonks for other large file challenges, I've seen references to sys::mmap and fork::parallelmanager. If anyone has used either of these (or others) and feel strongly about one, please let me know.