Hello, again. I apologize for the long absence, real life and such.

I extensively tested both Marshall's fork implementation and marioroy's MCE "entire-chunk" implementation and, depending on the test case, the results are very variable and very interesting.

Test case 1, a few thousands of <1MB files: fork.pl->2.2", mce.pl->9.4". The MCE one couldn't saturate my 4C/8T CPU, load was at around 60%.

Test case 2, the above files concatenated into a 30MB one: fork.pl->3.2", mce.pl->1.8". Here, the fork one left the CPU mostly idle.

Test case 3, 200 files sized 1.4MB each: fork.pl->15.2", mce.pl->25.7".

In general, MCE can't keep the CPU 100% busy, even with max_workers set at 2X the thread count. Also, chunk_size can make a big difference depending on the size of the files processed. For large single-file input, MCE simply dominates; for lots of small files, the fork implementation is the undisputed winner.

Most importantly, you people proved that Perl, in the right hands of course, can outperform almost everything and I'd like to thank every monk who shared his wisdom with me in this thread. My OP question has been fully answered.

In reply to Re: Need to speed up many regex substitutions and somehow make them a here-doc list by xnous
in thread Need to speed up many regex substitutions and somehow make them a here-doc list by xnous

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.