use strict;

for (1..10) {
    my $t0 = time();
    my $lines = read_log();
    print "Used ", time() - $t0, "\n"
}

sub read_log {
    open(FILE, "<", "test.txt");
    my @lines = <FILE>;
    close(FILE);
    return \@lines
}
[download]

Nope.

This code suffers from the same bug as the other code.

Notice that he said perl 5.6 on Win2k.

---

Updated

Also tested other solutions, with 5.6 on win98, this yielded the best performance.

Please to show your benchmarks.

Read more... (1208 Bytes)

Benchmark: running list_io, split_slurp, while_io, each for at least 5
+ CPU seconds...
   list_io:  5 wallclock secs ( 3.89 usr +  1.34 sys =  5.24 CPU) @ 17
+21.98/s (n=9018)
split_slurp:  4 wallclock secs ( 3.02 usr +  2.21 sys =  5.23 CPU) @ 2
+725.38/s (n=14251)
  while_io:  7 wallclock secs ( 3.64 usr +  1.38 sys =  5.02 CPU) @ 16
+29.26/s (n=8174)
              Rate    while_io     list_io split_slurp
while_io    1629/s          --         -5%        -40%
list_io     1722/s          6%          --        -37%
split_slurp 2725/s         67%         58%          --
[download]

Don't trying to be confused, and think local $/, read in and then split would improve performance.

In light of the evidence I think you will have to reconsider.

(As I will explain, whether you use scalar context has not much to do with performance.

Depends what you mean. Reading in smaller chunks at a time reduces memory overhead and can thus have a signifigant effect on run time.

On the other hand, keep in mind, split is not free, it requires to walk thru the whole string. As anyone has a c background would know, string operation does hurt performance a lot, especially this kind of operation that invloves head to toes.)

You would think this on face glance. As I said the evidence contradicts you.

Perhaps its due to perl being able to allocate one buffer sysread the lot and then walk the string. It may in fact be that this is more efficient than reading whatever the standard size buffer is for PerlIO, scaning it for new lines, then reading another buffer.... (Assuming of course memory is available)

First layer, the physical reading layer, Perl would read in block by block, doesn't matter whether your code requires a scalar context or array context. This makes sense, it is optimized for Perl's own performance.

By block by block presumably you mean buffer by buffer.

Second layer, the layer between Perl and your program. Perl would present the data in the right context, as you required. This layer doesn't involve physical devices, and is much less related to performance than the first layer does.

You have the return type part of context mixed up with the actions that the context causes to happen. In list context the IO operator does something different to what it does in scalar context. In list context it causes the entire file to be eventually read into mememory and sliced up into chunks as specified by $/.

In scalar context it reads enough to provide a single chunk at a time. If the chunks are small it may read more chunks into memory than one, but it doesn't necessarily load them all.

I will agree that I was suprised myself however about these results. But you cant just say that something works the way it does, and that it should be faster, because you think so. Unless you have poured over the PerlIO code and unless you have benchmarked the issue in question rather exhaustively you have no way to know howfast something is going to run in Perl.

--- demerphq
my friends call me, usually because I'm late....

However I do clearly see a positive sign here, your idea is changing, it is good for everyone to improve themselves thru discussions, and change ideas. I do, you do, everyone does, that's why we all love this site!!

--- demerphq
my friends call me, usually because I'm late....

thanks for your reply.


The file is only one 1.5 MB. the first function gets completes within a matter of seconds. Then the second function call takes a very long time to complete. I was wrong with I said that the program hangs... but the second and the third function calls take 5 minutes each to complete. This doesn't make sense to me... because if the first call took just 5 - 6 seconds why did the second and third take such a long amount of time?

My guess is that the code that you are actually using looks more like

sub read_file {
   my $file=shift;
   open my $fh,$file or die "$file : $!";
   my @lines=<$fh>;
   return \@lines;
}
[download]

In which case its storing the data in memory. Which is perhaps overflowing your available physical ram. When that happens the OS starts swapping the memory out to disk (obviously a slow operation), which if it happens enough leads to a condition called thrashing where the OS is basically just moving memory back and forth from the disk, and the time taken for the repeated swapping completely overwhelms the time your code takes. Causing your code to look like it hangs. Try using some memory monitoring tool, or do some program analyiss to see exactly how much memory you are actually using. Hashes for instance consume far far more memory than they look. As does careless array manipulation. For instance if you have

my @array;
$array[100_000_000]=1;
[download]

then perl will have to allocate sufficient RAM for all 100 million slots, even though they arent used.

Without seeing real code, (as i said I dont think what you posted properly reflects what you are doing) its hard to say for certain.

Regards,

UPDATE: Since you are on Win2k you should take advantage of the Task manager and the Profiler that come with the OS. (Start -> Settings -> Control Panel -> Administrative Tools -> Profiler)

--- demerphq
my friends call me, usually because I'm late....

test_pre.pl:
use strict;

open(DATA, ">", "test.dat");
foreach (1..$ARGV[0]) {
    print DATA "$_\n";
}
close(DATA);
[download]

test_slurp.pl:
use strict;

use constant SIZE => 10000;

my $t0 = time();
my $lines = read_log_s();
my @data = (0..SIZE - 1);
foreach my $line (@{$lines}) {
    $line += ($data[$line % SIZE] - $data[($line + 1) % SIZE]);
}
print "Used ", time() - $t0, "\n";

sub read_log_s {
    local $/;
    open(FILE, "<", "test.dat");
    my @lines = split /\n/, <FILE>;
    close(FILE);
    return \@lines;
}

test.pl:
use strict;

use constant SIZE => 10000;

my $t0 = time();
my @data = (0..SIZE - 1);
open(FILE, "<", "test.dat");
my $line;
while ($line = <FILE>) {
    $line += ($data[$line % SIZE] - $data[($line + 1) % SIZE]);
}
close(FILE);
print "Used ", time() - $t0, "\n";
[download]

perl -w test_pre.pl 100000