in reply to Faster Hash Slices

Do you need to retain the values outside of the loop? When processing 8 million lines, I wonder if you put all of that data into a hash, as that is likely to run out of memory.

If the parsed data is "healthy" - as in all lines have the same number of fields, neatly separated by single TAB characters - and you only need the fields *during the iteration*, you could consider using Text::CSV_XS in combination with bind_columns. That will re-use the assigned variables for each line and /might/ save a lot of time.

--8<--- use strict; use warnings; use Benchmark qw(cmpthese); my @data = ("aaaaaaaaaa") x 100; my $line = join "\t" => @data; my $data = join "\n" => map { $line } 0 .. 50000; use Text::CSV_XS; my $csv = Text::CSV_XS->new ({ sep_char => "\t", binary => 1, auto_dia +g => 1 }); sub bsplit { open my $fh, "<", \$data; while (<$fh>) { chomp; my @aa = split /\t/ => $_; } } # bsplit sub csvxs { open my $fh, "<", \$data; my @aa = ("") x scalar @data; $csv->bind_columns (\(@aa)); while ($csv->getline ($fh)) { } } # csvxs cmpthese (10, { "split" => \&bsplit, "csv_xs" => \&csvxs, }); -->8--- => Rate split csv_xs split 1.60/s -- -27% csv_xs 2.21/s 38% --

Note that the difference is *completely* depending on what your data looks like!

my @data = ("aaaaa") x 10; my $line = join "\t" => @data; my $data = join "\n" => map { $line } 0 .. 500000; => Rate csv_xs split csv_xs 1.17/s -- -13% split 1.34/s 14% -- ------------------------------------------------- my @data = ("aaaaa") x 500; my $line = join "\t" => @data; my $data = join "\n" => map { $line } 0 .. 50000; => s/iter split csv_xs split 2.93 -- -57% csv_xs 1.25 134% -- ------------------------------------------------- my @data = ("aaaaaaaaaaaaaaaaaaaaaaaaaaa") x 500; my $line = join "\t" => @data; my $data = join "\n" => map { $line } 0 .. 50000; => s/iter csv_xs split csv_xs 3.74 -- -5% split 3.55 5% --

Even running the same on a different machine might change the results. Benchmark if things get important, but bench where it matters: do not draw conclusions on a benchmark run on a 64bit non-threaded non-debugging perl on a 64core machine with 128Gb of RAM to have the same result on an AIX 5.2 with 32bit threaded debugging perl with only 1 CPU and 2 Gb of RAM available to the process.


Enjoy, Have FUN! H.Merijn