As noted -> at the end of the day you want throughput. Lots of small reads are much slower than several larger ones. You can see the progression - bigger read, faster throughput, but only up to a certain point.

[root@devel3 root]# cat reader.pl
#!/usr/bin/perl
open $fh, '/root/big.file' or die $!;
1 while read( $fh, $buf, $ENV{BLOCK_SIZE} );
close $fh;

[root@devel3 root]# ll big.file
-rw-r--r--    1 root     root     100000000 Dec 31 04:31 big.file

[root@devel3 root]# BLOCK_SIZE=8 strace -e read -c perl /root/reader.p
+l
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.166282           7     24426           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.166282                 24426           total
[root@devel3 root]# BLOCK_SIZE=64 strace -e read -c perl /root/reader.
+pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.182985           7     24426           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.182985                 24426           total
[root@devel3 root]# BLOCK_SIZE=1024 strace -e read -c perl /root/reade
+r.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.170449           7     24427           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.170449                 24427           total
[root@devel3 root]# BLOCK_SIZE=2048 strace -e read -c perl /root/reade
+r.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.169480           7     24427           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.169480                 24427           total
[root@devel3 root]# BLOCK_SIZE=4096 strace -e read -c perl /root/reade
+r.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.171735           7     24427           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.171735                 24427           total
[root@devel3 root]# BLOCK_SIZE=8192 strace -e read -c perl /root/reade
+r.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.106189           9     12220           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.106189                 12220           total
[root@devel3 root]# BLOCK_SIZE=16384 strace -e read -c perl /root/read
+er.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.094355          15      6116           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.094355                  6116           total
[root@devel3 root]# BLOCK_SIZE=32768 strace -e read -c perl /root/read
+er.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.081012          26      3064           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.081012                  3064           total
[root@devel3 root]# BLOCK_SIZE=65536 strace -e read -c perl /root/read
+er.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.078593          51      1538           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.078593                  1538           total
[root@devel3 root]# BLOCK_SIZE=131072 strace -e read -c perl /root/rea
+der.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.099163         128       775           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.099163                   775           total
[root@devel3 root]# BLOCK_SIZE=262144 strace -e read -c perl /root/rea
+der.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.141184         358       394           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.141184                   394           total
[root@devel3 root]# BLOCK_SIZE=524288 strace -e read -c perl /root/rea
+der.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.175145         863       203           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.175145                   203           total
[root@devel3 root]# BLOCK_SIZE=1048576 strace -e read -c perl /root/re
+ader.pl
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.175813        1628       108           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.175813                   108           total
[root@devel3 root]#
[download]

On this system RAID 5 SCSI you can see that the system buffer size is 4192 bytes. Over this figure you get bigger reads and as you can see the throughput increases. Somewhat suprisingly (given past testing) the optimal point on the system I used is 65K block size and you can see performance drop off after this point. As previously noted in an earlier post you need to test the exact code/OS/system combo to find the sweet spot that optimizes throughput.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.