in reply to Re^11: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)
in thread Optimizing with Caching vs. Parallelizing (MCE::Map)

Thanks, this is amazing work! How much work would it be to try each of the operations within the script some suitable number of times, on each PDL version (or at least isgood), in order to see if any of them stand out as the one that's got slower? (or conceivably it's across the board).

Immediate surmise: there have been updates to the badvalue detection functionality to allow NaN to be used as a badvalue in 2.040 and fixes to that in 2.064. Ergo, if you do have capacity to performance-test at least isgood and see that's what's slowed down, that will be at least possible to fix. One approach there would be to break out the "badvalue is NaN" branch out of the current broadcast loop into its own loop to avoid constantly checking a value that doesn't change within nearly all operations (the badvalue) to see if it's still (or still not) a NaN.

Replies are listed 'Best First'.
Re^13: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)
by marioroy (Prior) on May 22, 2022 at 00:39 UTC

    Greetings, etj. My initial post was simply heads up with regards to slowness detected. I left the test code for you; or if you like, for the PDL team. If you haven't already, you'd find interesting things using Devel::NYTProf. A lot of subroutines are noticeably slower compared to PDL 2.021. The test script provides a starting point.

[reply]
      "A lot of" - can you help me by naming one of them, and saying what data it was run on? I hope you can see that would make your observation even more helpful.

      For those interested in why I'm not digging into this instantly, it's because the PDL::OpenCV work is nearly done, apart from generating a C wrapper per OpenCV submodule - I can only do (at most, and on a good day) one thing at a time ;-)

[reply]
        Tools to actually help with this problem visible at Re^2: profiling XS routines. (edited as requested; one person said a hyperlink to https://perlmonks.org/?node_id=11144333 logs people out, which seems unlikely - however there are 6 different sites, and an id://blah link is canonically correct whichever one you're on, instead of linking to the wrong site)
[reply]

        I was hoping that a member of the PDL team can view the graph and compare between PDL 2.021 and recent.

[reply]

In Section Meditations