thanos1983 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

Recently I was introduced to diamond operator from I/O Operators, through the thread (How to amend character count.).

Well based on the answers I started wondering what is the best and fastest way to extract data from files with thousands of lines. So I was trying to Benchmark the two two fastest from my opinion ways (while() and grep) in iterating files.

I did not think it would be a problem to replicate the behavior but I end up with the script hanging and doing nothing. I was not able to understand why, until I created separate sub routines.

When each sub routine is executed individually and the script exits everything works fine, but when the subroutines are executed sequentially the script stays hanged.

I can understand that is based on the @ARG but I do not know how to tell to the script that that there is no more files to process by using explicitly the diamond operator <> and not individually the @ARG

Sample of code that can replicate my problem:

#!/usr/bin/perl use strict; use warnings; use Benchmark::Forking qw( timethese cmpthese ); # UnixOS # use Benchmark qw(:all) ; # WindowsOS sub test_grep { my @matched = grep { $_ =~ /AGGT/ } <>; if (@matched) { for(@matched) { chomp $_; print "Grep: " . length($_) . "\n";} } close ARGV if eof; } sub test_while { my $count; while (<>) { if ($_ =~ /^>hsa/){ chomp (my $line = <>); $count .= length($line); $count .= " "; } } continue { close ARGV if eof; # reset $. each file } print "While: " . $count . "\n"; } # test_grep(); # Works fine separate but not combined test_while(); # Works fine separate but not combined __DATA__ $ perl test.pl test.txt Grep: 107 Grep: 251 ^C $ perl test.pl test.txt While: 107 251 ^C __END__ my $results = timethese( 2, { 'Grep' => sub { }, 'While' => sub { }, }, 'none' ); cmpthese( $results );

Sample of test.txt file with data for replication purposes:

>hsa_circ_0000001|chr1:1080738-1080845-|None|None ATGGGGTTGGGTCAGCCGTGCGGTCAGGTCAGGTCGGCCATGAGGTCAGGTGGGGTCGGCCATGAAGGTG +GTGGGGGTCATGAGGTCACAAGGGGGTCGGCCATGTG >hsa_circ_0000002|chr1:1158623-1159348-|NM_016176|SDF4 GGTGGATGTGAACACTGACCGGAAGATCAGTGCCAAGGAGATGCAGCGCTGGATCATGGAGAAGACGGCC +GAGCACTTCCAGGAGGCCATGGAGGAGAGCAAGACACACTTCCGCGCCGTGGACCCTGACGGGGACGGT +CACGTGTCTTGGGACGAGTATAAGGTGAAGTTTTTGGCGAGTAAAGGCCATAGCGAGAAGGAGGTTGCC +GACGCCATCAGGCTCAACGAGGAACTCAAAGTGGATGAGGAAA

Does any one know how to approach this minor problem?

Update: With the help of tobyink the following code demonstrates the solution to my problem but also that while is faster than grep on my loop because of the for loop that I am using as extra. Thanks everyone again for their time and effort.

#!/usr/bin/perl use strict; use warnings; use Benchmark::Forking qw( timethese cmpthese ); # UnixOS # use Benchmark qw(:all) ; # WindowsOS my @preserved = @ARGV; sub test_grep { @ARGV = @preserved; # restore original @ARGV my @matched = grep { $_ =~ /AGGT/ } <>; my $count; if (@matched) { for(@matched) { chomp $_; $count .= length($_) . " ";} } close ARGV if eof; } sub test_while { @ARGV = @preserved; # restore original @ARGV my $count; while (<>) { if ($_ =~ /AGGT/){ chomp; $count .= length($_) . " "; } } continue { close ARGV if eof; # reset $. each file } } my $results = timethese(10000000, { Grep => \&test_grep, While => \&test_while }, 'none'); cmpthese( $results ); __DATA__ perl test.pl test.txt Rate Grep While Grep 144175/s -- -21% While 182882/s 27% --

Thanks in advance for your time and effort.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Replies are listed 'Best First'.
Re: Benchmark diamond operator
by tobyink (Canon) on May 09, 2017 at 14:46 UTC
    #!/usr/bin/perl use strict; use warnings; use Benchmark::Forking qw( timethese cmpthese ); # UnixOS # use Benchmark qw(:all) ; # WindowsOS my @preserved = @ARGV; sub test_grep { @ARGV = @preserved; # restore original @ARGV my @matched = grep { $_ =~ /AGGT/ } <>; if (@matched) { for(@matched) { chomp $_; print "Grep: " . length($_) . "\n";} } close ARGV if eof; } sub test_while { @ARGV = @preserved; # restore original @ARGV my $count; while (<>) { if ($_ =~ /^>hsa/){ chomp (my $line = <>); $count .= length($line); $count .= " "; } } continue { close ARGV if eof; # reset $. each file } print "While: " . $count . "\n"; } my $results = timethese(2, { Grep => \&test_grep, While => \&test_whil +e }, 'none'); cmpthese( $results );

      Hello tobyink,

      Thanks, worked perfectly. :D

      But share some light, why is it working with @ARGV = @preserved; # restore original @ARGV?

      Thanks again for your time and effort.

      Seeking for Perl wisdom...on the process of learning...not there...yet!

        You need to understand what the diamond operator does:

        • Does the operator have an active filehandle, if not, shift an argument off @ARGV and open it for input as the active filehandle or if @ARGV is empty, set the active filehandle to STDIN.
        • If called in list context, return all lines from the active filehandle.
        • If called in scalar context and the filehandle is at the end of the file, return undef.
        • Otherwise, return a single line from the filehandle.

        That's pretty much it. I might be forgetting some subtleties.

        The key bit, I've highlighted for you in bold. It uses @ARGV to figure out what file to read, and it alters that array!

        So the first time you read a file using the diamond operator, you read it fine. The second time you try to read the file, the filename is no longer on @ARGV, so the diamond operator reads from STDIN instead.

        > But share some light, why is it working with @ARGV = @preserved; # restore original @ARGV?

        I' dunno well your skills but if you have not understood what the venerable monk tobyink says below, and directly answering to your cited above assertion:

        the <> diamond operators consumes @ARGV so if you save a safe copy in @preserved you can refill @ARGV when you need ie first line of the two subs in the tobyink's example above.

        As said below the <> diamond can also feed @ARGV opening SDTIN as last resource, so the above works because it restores @ARGV original contents at the top of the two testing subs.

        L*

        There are no rules, there are no thumbs..
        Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
Re: Benchmark diamond operator
by davido (Cardinal) on May 09, 2017 at 14:47 UTC

    The <> diamond operator, acting upon @ARGV is going to be tricky to get right, if you want to re-read the same file multiple times to benchmark. Better off using open to manually act upon the filenames passed into @ARGV. That way you can re-open for each run, or (probably faster), open once, tell, and seek to reset between iterations.

    There are faster alternatives than <>, but if your drive is a mechanical drive, shifting over to SSD is the biggest win.


    Dave

      Hello davido,

      Alternatives than <> that are faster? Can you propose some so I could use them as future reference?

      Thank you for your time and effort.

      Seeking for Perl wisdom...on the process of learning...not there...yet!

        <> is a general purpose tool, and as such is optimized for "the general case." There isn't one single tool that can optimize for speed across the general case (or in other words, the majority of all use cases). If you know your data-set well, you can optimize by selecting a less general tool to fit your specific need. Tools to consider include:

        • read: May be faster for small reads.
        • sysread: May be faster for large reads.
        • Sys::Mmap and File::Map: May be faster for large files in some specific cases.
        • Lower-impact layers (eg :raw or :mmap) may offer speed advantages in some cases while reducing functionality in other areas.

        Each of these methods come with caveats, pitfalls, limitations, and unique benefits. It is up to you to decide if the limitations are outweighed by the benefits for your specific use case. benchmarking is the only way you'll be sure that one is a win over the other for your specific situation.


        Dave

Re: Benchmark diamond operator
by 1nickt (Canon) on May 09, 2017 at 14:28 UTC

    Hi, if I understand you correctly, the issue is that the diamond operator uses shift to iterate through @ARGV, so the second sub will find it empty.

    $ perl -Mstrict -Mwarnings -E 'say "starting: " . scalar @ARGV; while( +<>){chomp; say "$_\nremaining: " . scalar @ARGV}' foo bar baz
    starting: 3 foo remaining: 2 bar remaining: 1 baz remaining: 0

    Hope this helps!

    Update: added demo


    The way forward always starts with a minimal test.

      Hello 1nickt ,

      I assume the same, but how can I make my script to stop after parsing the first input? I want to avoid using @ARGV but this would be my last option.

      Any ideas? Thanks for your time and effort.

      Seeking for Perl wisdom...on the process of learning...not there...yet!
Re: Benchmark diamond operator
by AnomalousMonk (Archbishop) on May 09, 2017 at 14:28 UTC
    close ARGV if eof;

    I haven't tested any of this, but I believe this close statement in each of the test_* functions will prevent the other from doing anything: you can't read anything on a closed filehandle. Rather than close-ing the ARGV filehandle, I would seek to the beginning of the handle before each data-read operation in each test function.

    Update:

    ... seek to the beginning of the handle before each data-read operation in each test function.
    A little experimentation shows my idea is wrong. seek cannot be used on ARGV before execution of the  <> operator in the
        while (<>) { ... }
    loop because the handle has never been opened. seek cannot be used on ARGV after the loop because the handle has been closed. It's possible to seek, etc., on ARGV within the loop, but I can't see how this would be useful in this particular situation.


    Give a man a fish:  <%-{-{-{-<

      Hello AnomalousMonk,

      Well I thought that closing the file would allow to open the file again on next iteration. I could try to apply what you proposed and see how it goes.

      Thanks for your time and effort.

      Seeking for Perl wisdom...on the process of learning...not there...yet!