andye has asked for the wisdom of the Perl Monks concerning the following question:

Hi folks,

I'm using PDL (the Perl Data Language, pdl.perl.org) to manipulate a Very Large dataset.

I'm having quite a bit of trouble with the correct syntax for threading, can't seem to get it right.

Right now I'm doing a lookup, where an old value gets turned into a new value. There's a one-to-one relationship between old values and new values. Currently I'm doing this using a normal hash, like this: 

    my @old_values = list($pdl_oldvalues);
    my @new_values = map $hash_lookup->{$_}, @old_values; 
    my $pdl_newvalues = pdl(\@new_values);

[download]

I'd like to do this using PDL threading if I can, because I want it to go quicker (there are a lot of values), but I can't find the right way to do it.

I've tried: 

    my $pdl_newvals = $hash_lookup->{$pdl_oldvals};

[download]
but sadly that doesn't work. I've tried using $pdl_oldvals->thread(0) and $pdl_oldvals->dummy(0) but they don't work either.

I've also tried turning the hash into a two-row pdl, with keys on one row and values on the other row, then taking a slice from that pdl to get the values. That should certainly work with pdl threading... but I can't get it to. :(

Suggestions, and examples of correct syntax for doing this kind of thing in pdl, very much appreciated.

All the best, andye.

PS where "doesn't work" = either a syntax error, or runs but produces the wrong values...

Replies are listed 'Best First'.
Re: Hash-type lookup in PDL
by zentara (Cardinal) on Jan 05, 2007 at 17:28 UTC
    Could you possibly work up a simple running script, with sample data, so we can see what format your data is in, and the dimensions of your piddles? It sounds to me my you are looking for "slices", to do the changes in place (efficiently), but I have no idea what your piddle looks like.

    The cool thing about slices, is that whatever operation you perform on the slice, also occurs to the original piddle. 

    #!/usr/bin/perl
use warnings;
use strict;
use PDL::LiteF;


my $x = sequence(25);  # Create a piddle of increasing value
print "$x\n";                

my $slice =  slice $x,'4:20:2';
print $slice;
print "\n";

$slice .= 0;
print $slice;
print "\n";

print $x;
print "\n";

for (1..100){
$slice ++;
print $x;
print "\n";
select(undef,undef,undef,.5);
}

    [download]
    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
[reply]
[d/l]
      Hi zentara, It's nothing complicated I'm looking to do, really, just lookup each value in a hash. Here's an example that doesn't work: 
      #!/usr/bin/perl

use warnings;
use strict;
use PDL;

my $x = sequence(5); print "$x \n";

my $lookup = {     
                0 => 101,
                1 => 69,
                2 => 42,
                3 => 10042,
                4 => 99
              };

my $y = zeroes(5); $y .= $lookup->{$x}; print "$y \n";

      [download]
      That gives this output: 
      [0 1 2 3 4] 
[0 0 0 0 0]

      [download]
      I would be a happy bunny if it gave this output: 
      [0 1 2 3 4] 
[101 69 42 10042 99]

      [download]
      Best wishes, andye

      (I tried using a straightforward my $y = $lookup->{$x} but that doesn't produce any value at all)

      update: I've tried creating a pdl to use insread of the hash, like this: 

      my $pdl_table = pdl([keys %$lookup], [values %$lookup]);

my $y = zeroes(5); $y .= $pdl_table->(which($pdl_table->(:,0) == $x),1
+); print "$y \n";

      [download]
      but that has two problems:
      • It's rather hard to read
      • It doesn't work
      Looks like I've screwed up the syntax in the which() statement, and I can't work out how to make it right - a dummy dimension somewhere?
[reply]
[d/l]
[select]

        Hi andye,

        If your lookup table is rather small, you could try this code:

        #!/usr/bin/perl

use warnings;
use strict;
use PDL;

my $x = sequence(5); print "$x \n";

my @pdl_keys   =     (0,    1,   2,      3,   4);
my @pdl_values =     (101, 69,  42,  10042,  99);

my $y = zeroes(5);
foreach my $i (0..@pdl_keys-1){
    my $indx = which($x == $pdl_keys[$i]);
    $y->dice($indx) .= $pdl_values[$i];
}
print "$y \n";

        [download]

        I hope this helps

        Cheers,

        lin0
[reply]
[d/l]
Re: Hash-type lookup in PDL
by zentara (Cardinal) on Jan 05, 2007 at 22:03 UTC
    Here are a couple of other ideas for you. 
    #!/usr/bin/perl
use warnings;
use strict;
use PDL;

my $x = sequence(5); print "$x \n";

my $lookup = {     
                0 => 101,
                1 => 69,
                2 => 42,
                3 => 10042,
                4 => 99
              };

my $y = zeroes(5); 

foreach (keys %{$lookup}){ 
      set $y, ,$_ ,$lookup->{$_};
 }

print "$y \n";

    [download]

    This one is probably more useful, because you can read in a large amount of data. 

    #!/usr/bin/perl
use warnings;
use strict;
use PDL;
use PDL::IO::Misc;

my $z  = rcols *DATA;
print "$z\n";

print $z->at(1);
print "\n";

__DATA__
101 
69 
42 
10042 
99

    [download]
    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom