in reply to Re^7: Faster and more efficient way to read a file vertically
in thread Faster and more efficient way to read a file vertically

Hi, johngg, I rather tried to communicate that all contestants should be placed in equal conditions, i.e. let them all read in chunks, rather than in single lines, as it was for some of them. But all the same your "ANDmask" is fastest.

Which is weird. Simple act of extracting some part of data, instead of just indexing, requires, to be fast, modification of unrelated data. Sure, it's because of speed of "anding" and transliteration, but still... Therefore, here is something completely different (sorry I keep adjusting your setup to suite my "dna.txt"):

use strict; use warnings; use Benchmark qw{ cmpthese timethese }; use Test::More qw{ no_plan }; use String::Random 'random_regex'; my $fn = 'dna.txt'; unless ( -e $fn ) { open my $fh, '>', $fn; print $fh random_regex( '[ACTG]{42}' ), "\n" for 1 .. 1e6; } open my $inFH, q{<}, $fn or die $!; binmode $inFH; my $buffer = <$inFH>; my $lineLen = length $buffer; my $nLines = 500; my $chunkSize = $lineLen * $nLines; my $offset = 9; # Column 10 if numbering from 1 my %methods = ( ANDmask => sub { # Multi-line AND mask by johngg seek $inFH, 0, 0; my $retStr; my $mask = qq{\x00} x ${offset} . qq{\xff} . qq{\x00} x ( $lineLen - $offset - 1 ); $mask x= $nLines; while ( my $bytesRead = read $inFH, $buffer, $chunkSize ) { ( my $anded = $buffer & $mask ) =~ tr{\x00}{}d; $retStr .= $anded; } return \ $retStr; }, pdl => sub { seek $inFH, 0, 0; use PDL; my $retStr; my $chunkPDL = zeroes( byte, $lineLen, $nLines ); my $bufRef = $chunkPDL-> get_dataref; while ( my $bytesRead = read $inFH, $$bufRef, $chunkSize ) { my $lastLine = $bytesRead / $lineLen - 1; $retStr .= ${ $chunkPDL-> slice( "$offset,0:$lastLine" ) -> get_dataref } } return \ $retStr; }, ); ok ${ $methods{ ANDmask }-> ()} eq ${ $methods{ pdl }-> ()}; cmpthese( -10, { map { $_ => $methods{ $_ } } keys %methods });

>perl vert5.pl ok 1 Rate ANDmask pdl ANDmask 7.86/s -- -55% pdl 17.3/s 120% -- 1..1

Replies are listed 'Best First'.
Re^9: Faster and more efficient way to read a file vertically
by johngg (Canon) on Nov 09, 2017 at 11:33 UTC

    Interesting! I don't have PDL installed on my system but, reading about it, it seems that it contains highly optimised routines for array handling largely written in C. This would probably account for the increase in speed. I'm not a geneticist so I don't know if the OP's problem is a real-world scenario but I wonder if PDL has been used widely in the field. Thank you for pointing it out.

    Cheers,

    JohnGG

      I believe PDL is somewhat used in genetics, as evidenced by PDL::HMM et al. It's interesting that there doesn't seem to be much use of PDL in BioPerl.