in reply to Re: Poor performances with threads
in thread Poor performances with threads

One can calculate the standard deviation with just one pass over the data set by iterating once over the numbers, calculating both the sum and the sum of squares in one go.

In the case of only 7 numbers to sum over that doesn't make much of a difference (0.57s vs. 0.48s on my machine), The difference becomes more pronounced when there are more numbers in one set, when the whole array doesn't fit into the cache anymore (2.94s vs. 4.04s for 1k lists with 7k items each).

But when such micro optimizations become important, it's worth pointing out that writing things in C (via XS or Inline::C) might be much faster.

Perl 6 - links to (nearly) everything that is Perl 6.

Replies are listed 'Best First'.
Re^3: Poor performances with threads
by BrowserUk (Patriarch) on Oct 23, 2009 at 12:06 UTC
    doesn't make much of a difference (0.57s vs. 0.48s on my machine)

    Agreed. But once you've reduced the time from 60 seconds to 0.6 of a second, the extra saving hardly seems worth it :) But I guess that's up to the OP.

    But it does highlight the problems with modules like Statistics::Basic. It offers a OO interface, storing copies of the data internally with the intention that it only need calculate things like mean once, whilst providing access to many types of statistic which re-use that calculated value.

    The problem is, calculating the mean of 7 values manually is quicker than calling a nested inherited method to retrieve it from storage. If you are processing single large datasets, amortisations can lead to savings (assuming you actually use the OO interface which the OP isn't). But for processing many small datasets, there are no amortisation gains.

    The upshot is, users pay the penalties of the OO without gaining any benefits and then go off looking at complex solutions (threads in this case) to try and recover the losses.

    So, 41 installed files and 3 dozen lines of code (and more), to achieve what can be done with 6 lines much more productively.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom