Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Boosting workaround [Was: PDL's error or mine?]

by jo37 (Deacon)
on Feb 23, 2023 at 11:33 UTC ( [id://11150550]=note: print w/replies, xml ) Need Help??


in reply to Re: PDL's error or mine?
in thread PDL's error or mine?

Opened an issue as suggested, including the strange behaviour as discovered by choroba

Interestingly, the search for a suitable workaround resulted in a remarkable boost for my specific application. Here the sets represent indices from another piddle and one operand is taken from a piddle holding several sets, which requires equal dimensions for all sets. This is achieved by padding the sets with BAD values at the end. With a small modification, the values can be placed just at the position they represent and the remaining values set to BAD. Then an intersection may be performed almost with a simple dice.

E.g.

my $data = pdl ...; # 1-d #my $set1 = $data->where(something); # before Update 1 #my $set1 = $data->which(something); # before Update 2 my $set1 = which($mask1); # $mask1 having the shape of $data my $full1 = zeroes(indx, $data->dim(0))->setvaltobad(0); $full1->dice($set1) .= $set1; #my $set2 = $data->where(other); # before Update 1 #my $set2 = $data->which(other); # before Update 2 my $set2 = which($mask2); # $mask2 having the shape of $data # Now the intersection of $set1 and $set2 is: my $tmp = $full1->dice($set2)->sever; my $intersect = $tmp->where(isgood $tmp);
The result in $intersect can be found much faster with dice than with setops for a large sized $data. An older issue with setops revealed the usage of uniq within setops, which sorts the data making it O(N log N), while the dice approch should be O(N). In my application there is no overhead in constructing $full1.

Probably this is already described somewhere else.

Update 1:
It must be $data->which(...) instead of $data->where(...). Modified the example.

Update 2:
The example was still faulty. The key for this to work is $set1 and $set2 being piddles holding indices from $data as said in the paragraph above. Modified the example once again. Regard it as pseudocode.

Greetings,
-jo

$gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$

Replies are listed 'Best First'.
Re: Boosting workaround [Was: PDL's error or mine?]
by etj (Deacon) on Feb 28, 2023 at 02:03 UTC
    This has been fixed in PDL and (so far) dev-released as 2.081_01. Thank you!
Re: Boosting workaround [Was: PDL's error or mine?]
by Anonymous Monk on Feb 23, 2023 at 12:26 UTC

    Wouldn't "dice" fail if $set1->maximum > $data->dim(0) ? (or even s/set1/data/ in general?)

      You are perfectly right because I had a typo in my example. It should read ->which(...) instead of ->where(...). In that case, the condition cannot be violated.

      Greetings,
      -jo

      $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$

        OK... It was un-runnable pseudo-code anyway; and which as function expects single argument; and, as a method, should be called on a mask without other arguments, right?

        Even then, I'm confused, wouldn't you get exactly same result (i.e. $intersect) by simply and-ing ("&") the 2 masks and calling which on it? If example was just simplified for laymen to this tautology, I certainly expect no explanation, sorry to bother.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11150550]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (2)
As of 2024-04-26 03:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found