Re^2: Mysterious slow down with large data set

Yes, I can't believe I did that. Thank you!

Regarding the top N, I had previously tried this, which worked, but seemed more complicated. Is this what you had in mind?

    } elsif ($sim > min(pdl(@topList))) { 
      $theMin = grep { $topX[$_] eq min(pdl(@topX)) } 0..$#topX;
      # replace the smallest
      $topX[$theMin] = $sim;
      # add this one
      push @topX, $sim;
}
[download]

Thanks!

Comment on Re^2: Mysterious slow down with large data set Download Code

Replies are listed 'Best First'.
Re^3: Mysterious slow down with large data set by BrowserUk (Patriarch) on Feb 27, 2012 at 00:22 UTC
I tried it this way: `@topX = (-1) x 20; ... $topX[ $_ ] < $sim and splice( @topX, $_, 0, $sim ), pop( @top +X ), last for 0 .. 19;` [download] A short-ciruited, linear insertion is at worst O(N) rather than O(N logN). It speeds things a little, but doesn't address the slowdown which is happening exclusively (and inexplicably) inside PDL. Unfortunately, the PDL documentation spends more time telling you about their 'philosophy'; and indexing the indexes to the documentation than is does telling you what these functions actually do; or how to inspect the results of what they did :( With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Mysterious slow down with large data set
by BrowserUk (Patriarch) on Feb 27, 2012 at 00:22 UTC

I tried it this way:

    @topX = (-1) x 20;
...
        $topX[ $_ ] < $sim and splice( @topX, $_, 0, $sim ), pop( @top
+X ), last for 0 .. 19;
[download]

A short-ciruited, linear insertion is at worst O(N) rather than O(N logN).

It speeds things a little, but doesn't address the slowdown which is happening exclusively (and inexplicably) inside PDL.

Unfortunately, the PDL documentation spends more time telling you about their 'philosophy'; and indexing the indexes to the documentation than is does telling you what these functions actually do; or how to inspect the results of what they did :(

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

[reply]
[d/l]