The following is a parallel demonstration, based on the code by BrowserUk. There is a bug in MCE::Shared 1.001 and the reason for the length check below. MCE 1.704 and MCE 1.002 will be released in ETA ~ 1 week with the fix.

MCE::Flow and MCE::Shared

use strict;
use warnings;

use MCE::Flow;
use MCE::Shared;
use Time::HiRes qw[ time ];

my $start = time;
my $fh = MCE::Shared->handle( "<:raw", $ARGV[ 0 ] );
my @seen;

sub tally {
   my ($aref) = @_;
   for ( 0 .. 255 ) {
      $seen[$_] += $aref->[$_] if $aref->[$_];
   }
   return;
}

mce_flow { max_workers => 8 }, sub {
   my @_seen;
   while( read( $fh, my $buf, 16384 * 4 ) ) {
      # the length check may be omitted with MCE::Shared 1.002+
      last unless length($buf);
      ++$_seen[$_] for unpack 'C*', $buf;
   }
   MCE->do('tally', \@_seen);
};

close $fh;

printf "Took %f secs\n", time() - $start;

# for ( 0 .. 255 ) {
#    printf "%c : %u\n", $_, $seen[$_] if $seen[$_];
# }

[download]

The serial code takes 8.390 seconds. In comparison, the parallel code completes in 2.253 seconds for a 126 MB file on a machine with 4 real cores and 4 hyper-threads.

Update:

The upcoming MCE::Shared 1.002 release will support the following construction by allowing the main or worker process to handle the error. I've been wanting for the shared open call to feel like the native open call.

use MCE::Shared 1.002;

mce_open my $IN, "<:gzip", "wat.paths.gz" or die "open error: $!";
mce_open my $OUT, ">", \*STDOUT or die "open error: $!";

[download]

In reply to Re: Count byte/character occurrence (quickly) by marioroy
in thread Count byte/character occurrence (quickly) by james28909

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.