in reply to Performance In Perl

How often will you be running this code?

Once?

Daily?

Even if you run it once a second, 24/7, for a year, the performance difference is likely to be so small that the total time saved over that entire year will be less than the time it took you to type out your question.

Optimizing for programmer time is generally far more effective than trying to micro-optimize for CPU time.

Replies are listed 'Best First'.
Re^2: Performance In Perl
by Mano_Man (Acolyte) on Aug 28, 2017 at 13:54 UTC
    I do not agree. A code habit is a code habit, and is installed once. If you are used to writing scripts that intend to print out large bulks of data , let's say, a 700MB file, and do so multiple times a day, a 500% increase in performance as found a few comments back might well indeed be substantial. What about if multiple users use those scripts ? Let's say, a 100 people, as is the problem in my case. Would your answer change ?
[reply]
      In this particular case, no, it would not change. "A 500% increase in performance" is a meaningless statistic without first identifying 500% of what and, in this case, that base number is too tiny to matter, especially in comparison to the time your program will need to spend reading the 700M file from disk and doing the actual processing of its contents before it's ready to print its output.

      Also, while kcott's numbers above show a 981% difference (roughly 0.2ms vs 2ms for 20k lines of output, which is to say a fraction of a microsecond per line), I note that his test builds the long strings using the x operator instead of doing 20k individual concatenations. Let's see what happens if we actually build the output string line-by-line instead, as the code in your original post does it: 

      #!/usr/bin/env perl

use strict;
use warnings;
use autodie;

use constant {
    LINES   => 20_000,
    RECORD  => 'X' x 100 . "\n",
}; 

use Benchmark 'cmpthese';

open my $fh, '>>', '/dev/null';
my $out;

cmpthese 0 => {
    kcott_by_line => sub {
        print $fh RECORD for 1 .. LINES;
    },
    kcott_concat => sub {
        print $fh RECORD x LINES;
    },
    append_per_line => sub {
      $out = '';
      $out .= RECORD for 1 .. LINES;
      print $fh $out;
    }
}

      [download]
      And the results: 
                        Rate   kcott_by_line append_per_line    kcott_concat
kcott_by_line    716/s              --            -20%            -93%
append_per_line  900/s             26%              --            -91%
kcott_concat    9690/s           1254%            976%              --

      [download]
      Only a 26% difference between printing line-by-line and appending line-by-line. It seems that the primary optimization behind a single print being so much faster in kcott's test was that it built the entire output string in one operation instead of handling each line of output separately. Which is not an optimization that you would be able to apply in the case your question describes.

      And, again translating this back into real numbers, the difference is 14.3 million lines/second printing them individually vs. an even 18 million lines/sec if they're concatenated first. 0.7 microseconds/line vs. 0.56 microseconds/line. A savings of approximately one second per 70 million lines of output. Over four billion lines to get a one-minute difference.

      Whoopty-freaking-do.

      How many times would each of those 100 users have to process their 700M input files for the aggregate difference to add up to the time you spent reading this reply, never mind the time I spent writing it?

      This kind of micro-optimization is just not worth it in 99% of cases - and, for the other 1%, you'll get bigger gains by using C or a similar high-performance language instead of Perl, and then micro-optimizing the C code if you still need more speed at that point.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom