jo37 has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks and nuns,

when playing with PDL's range function I stumbled across a problem. While for dimensions up to 3 everything looks good, at d=4 something seems to be seriously broken. There is some memory corruption causing a segfault.

Looking at d=3 everything looks as expected: 

my $p = zeroes(4,4,4);
say 'p:        ', $p->info;
say 'ndcoords: ', ndcoords($p)->info;
say 'range:    ', $p->range(ndcoords($p), 2, 't')->info;

[download]

produces: 

p:        PDL: Double D [4,4,4]
ndcoords: PDL: Double D [3,4,4,4]
range:    PDL: Double D [4,4,4,2,2,2]

[download]

The first dimensions of range correspond to the second to last dimensions of the index piddle, i.e. [4,4,4]. The following dimensions correspond to the shape of the selected slices, i.e. [2,2,2].

With d=4 things become weird. 

my $q = zeroes(5,5,5,5);
say 'q:        ', $q->info;
say 'ndcoords: ', ndcoords($q)->info;
say 'range:    ', $q->range(ndcoords($q), 2, 't')->info;

[download]

produces: 

q:        PDL: Double D [5,5,5,5]
ndcoords: PDL: Double D [4,5,5,5,5]
range:    PDL: Double D [5,5,5,5,2,2,1,5]

[download]

Here we have - as expected - the second to last dimensions of the index piddle coming first. But then, instead of the expected [2,2,2,2] remainder, there comes [2,2,1,5]. Memory is corrupted afterwards, as it appears here on my Debian 11. So here is the full example script: 

#!/usr/bin/perl

use v5.24;
use warnings;
use PDL;

my $p = zeroes(4,4,4);
say 'p:        ', $p->info;
say 'ndcoords: ', ndcoords($p)->info;
say 'range:    ', $p->range(ndcoords($p), 2, 't')->info;
say ones(4);

my $q = zeroes(5,5,5,5);
say 'q:        ', $q->info;
say 'ndcoords: ', ndcoords($q)->info;
say 'range:    ', $q->range(ndcoords($q), 2, 't')->info;
say ones(5);

__DATA__
p:        PDL: Double D [4,4,4]
ndcoords: PDL: Double D [3,4,4,4]
range:    PDL: Double D [4,4,4,2,2,2]
[1 1 1 1]
q:        PDL: Double D [5,5,5,5]
ndcoords: PDL: Double D [4,5,5,5,5]
range:    PDL: Double D [5,5,5,5,2,2,1,5]
Segmentation fault

[download]

Can anybody reproduce this behaviour? If so, I'd report a bug.

Greetings,
-jo

$gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$

Replies are listed 'Best First'.
Re: PDL range on 4-d piddles
by choroba (Cardinal) on Oct 19, 2023 at 18:05 UTC
    What version of PDL do you have? Mine (2.025) was crashing with the segmentation fault, too, but after an upgrade to 2.084, I'm getting the following output: 
    p:        PDL: Double D [4,4,4]
ndcoords: PDL: Double D [3,4,4,4]
range:    PDL: Double D [4,4,4,2,2,2]
[1 1 1 1]
q:        PDL: Double D [5,5,5,5]
ndcoords: PDL: Double D [4,5,5,5,5]
range:    PDL: Double D [5,5,5,5,2,2,1,5]
[1 1 1 1 1]

    [download]
    The 2, 2, 1, 5 is still there, but there are no errors.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
[select]

      Mine is 2.025, too. I'll upgrade and try some non-trivial tests. Doesn't look sane, though.

      Thanks for testing!

      Greetings,
      -jo

      $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$
[reply]
Re: PDL range on 4-d piddles
by jo37 (Curate) on Oct 19, 2023 at 20:14 UTC

    As suggested by choroba, I upgraded to 2.084. It doesn't crash anymore, but it does not seem to work right. I have a little problem with 4-d piddles, though - they are just a bit beyond my imagination. However, there are things where you just can feel they are wrong. So let's take a 5x5x5x5 piddle filled with the sequence 0..(5^4-1). From this piddle we take a 2x2x2x2 slice around every piddle element and take take the sum over the slice's elements. Obviously, no such sum can ever be less than 28 as the sum of 0..7. However: 

    #!/usr/bin/perl

use v5.24;
use warnings;
use PDL;

my $q = sequence(5,5,5,5);
say 'q:          ', $q->info;
say 'ndcoords:   ', ndcoords($q)->info;
say 'range:      ', $q->range(ndcoords($q), 2, 't')->info;
say '2x2x2x2 sum:', $q->range(ndcoords($q), 2, 't')->reorder(4,5,6,7,0
+,1,2,3)->clump(4)->sumover;

__DATA__
q:          PDL: Double D [5,5,5,5]
ndcoords:   PDL: Double D [4,5,5,5,5]
range:      PDL: Double D [5,5,5,5,2,2,1,5]
2x2x2x2 sum:
[
 [
  [
   [  0   1   2   3   4]
   [  5   6   7 588 296]
   [604 612 620 628 316]
   [644 652 660 668 336]
   [332 336 340 344 173]
  ]
  [
   [724 732 740 748 376]
   [764 772 780 788 396]
   [804 812 820 828 416]
   [844 852 860 868 436]
   [432 436 440 444 223]
  ]

...

    [download]

    Update:

    The problem is much easier to see if we use a all-ones piddle and take periodic boundaries. Then all slices are made of ones and the sum over a 2^D slice is (or: should be) always 2^D.

    However: 

    #!/usr/bin/perl

use v5.24;
use warnings;
use PDL;

my $q = ones(5,5,5,5);
say 'q:          ', $q->info;
say 'ndcoords:   ', ndcoords($q)->info;
say 'range:      ', $q->range(ndcoords($q), 2, 'p')->info;
say '2x2x2x2 sum:', $q->range(ndcoords($q), 2, 'p')->reorder(4,5,6,7,0
+,1,2,3)->clump(4)->sumover;

__DATA__
[
 [
  [
   [9 9 9 9 9]
   [9 9 9 8 8]
   [8 8 8 8 8]
   [8 8 8 8 8]
   [8 8 8 8 8]
  ]
  [
   [8 8 8 8 8]
   [8 8 8 8 8]
   [8 8 8 8 8]
   [8 8 8 8 8]
   [8 8 8 8 8]
  ]
...

    [download]

    Greetings,
    -jo

    $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$
[reply]
[d/l]
[select]
      Thanks to all for their investigation, and also for reporting this!

      This is indeed a fascinating problem. The reason why the higher-dim output ndarrays have dims that start with 1, then are powers of the original size, is because the pdl struct has a def_dims array, size 6. Straight after that is the def_dimincs (stride-size at each dimension), also size 6 (the API has code to allocate memory if >6 dims, clearly that's not getting called here). Both are being used, even though there are more than 6 dims. Reading the array of dims will therefore give data that starts out correct, then looks crackers. It's almost certainly a bug in the rangeb code, since higher-dim ndarrays surely get tested elsewhere and the API ought to be solid.

      Further discussion etc on the GitHub issue, but I thought I'd give an insight here into what's going on.

[reply]
[d/l]
[select]
        The problem is now fixed (in the git repo), by deleting one line. See the GitHub issue for more details!
[reply]
Re: PDL range on 4-d piddles
by jo37 (Curate) on Oct 20, 2023 at 16:10 UTC
[reply]
Re: PDL range on 4-d piddles
by Anonymous Monk on Oct 20, 2023 at 10:59 UTC

    That's fun. 1st loop is tautology test for a default value of 1 for size, which (tautology) I think you are aware of, and which (test) suddenly fails after 6. 2nd loop causes Perl to crash after 6, but just manages to produce weird results for lower dimensions, too.

    use strict; 
use warnings;
use feature 'say';
use PDL;
use Test::PDL 'eq_pdl';

say $^V;
say $PDL::VERSION;

my @a = map sequence(( 5 ) x $_ ), 1 .. 8;

printf "%d ---> %s\n", 
    $_-> ndims, 
    eq_pdl( $_, $_-> range( $_-> ndcoords )) ? 'pass' : 'fail'
for @a; 

printf "%d ---> %s\n", 
    $_-> ndims, 
    $_-> range( $_-> ndcoords, 2, 't' )-> shape
for @a; 

__END__

v5.32.1
2.084
1 ---> pass
2 ---> pass
3 ---> pass
4 ---> pass
5 ---> pass
6 ---> pass
7 ---> fail
8 ---> fail
1 ---> [5 2]
2 ---> [5 5 2 2]
3 ---> [5 5 5 2 2 2]
4 ---> [5 5 5 5 2 2 1 5]
5 ---> [5 5 5 5 5 2 1 5 25 125]
6 ---> [5 5 5 5 5 5 1 5 25 125 625 3125]

    [download]
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom