in reply to Re^4: Fast algorithm for 2d array queries (Got OpenGL::Array)
in thread Fast algorithm for 2d array queries

Hmm, [ some XXX module ] might speed it up ?

At the very minimum, the code would need to increment an integer (one of many) , 500 * 50000 times. That takes at least 1.6 seconds in Perl:

$t=time; 
$i=0; 
++$i for 1 .. 25e6; 
print time-$t;;
1.66395616531372

[download]

That 7000 times too slow for the OPs target rate.

And that's before you take into account indexing into the arrays; indexing into the counts; and scanning the final tallies to determine which count is highest.

This is impossible using a single machine, even if written in highly optimised C or assembler.

It might just be possible if you can throw enough hardware at the problem such that you could pre-index every possible combination. Maybe.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: Fast algorithm for 2d array queries (GPU OpenGL::Array)
by Anonymous Monk on Feb 07, 2014 at 21:19 UTC

    This is impossible using a single machine, even if written in highly optimised C or assembler.

    I buy that argument for CPU, but GPU works in parallel, so it should be faster -- maybe not fast enough for desired target rate, but certainly to knock down the number of machines required by at least 1000 at minimum :)

[reply]
      but GPU works in parallel, so it should be faster

      Only if the required algorithm can be coded to make use of the inherent concurrency of the GPU instruction set.

      And (without expending great effort into looking at what instructions are available) I see three problems:

      1. Instructions applicable to the task.

        GPU are great at applying a single operation, or a restricted set of pairs of operations (multiply & add) to multiple values concurrently.

        But unless a way can be devised to perform a boolean test & increment instruction using the usually mathematical instruction set; it's not gonna fly.

      2. Integers versus float/double operands.

        Most GPU instructions deal with floating point values. The OPs problem revolves around integer values and boolean operations. If the data has to be converted first, if might kill any gains.

      3. Memory transfers.

        Shipping large volumes of data from cpu to gpu memory and back can be a major sap on performance and throughput.

      I'm not saying it isn't possible. I am saying that it might take considerable R&D to make it work.

      Better, if you know how to do it, I'd love to learn :)

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom