How to dump a PDL

llancet has asked for the wisdom of the Perl Monks concerning the following question:

I made an artificial neural network module, in which lots of piddles are used to store the connection weights. I need to store those weights to store the network state. Currently, I convert piddles into perl arrays using following way:

my @scalar;
my $size = $piddle->getdim(0);
for (my $i=0; $i<$size; $i++) {
  push @scalar,$piddle->at($i)
}
[download]

and dump a very large perl AOA into string using YAML.
However, this implementation is very slow, especially when there are ten thousands of piddles to work through. So, how can I dump them with high efficiency?

Comment on How to dump a PDL Download Code

Replies are listed 'Best First'.
Re: How to dump a PDL by Anonymous Monk on Dec 26, 2010 at 02:20 UTC
PDL::IO::Storable?	[reply]
Re: How to dump a PDL by trwww (Priest) on Dec 26, 2010 at 03:09 UTC
Without more info, my guesses would be a SQL database with a solid state drive, or an in memory SQL database. Maybe the data would map well to one of those newfangled NoSQL databases? Regardless, its not surprising that serializing large data structured to text formats and back is slow.	[reply]
Re: How to dump a PDL by zentara (Cardinal) on Dec 26, 2010 at 23:34 UTC
Are you looking for dog? From my old notes: `#!/usr/bin/perl use warnings; use strict; use PDL; # use dog to write 2-d piddles, both stdout or to a file. my $pdl = random( 1000, 1000); wcols "%6.1f",$pdl->dog;` [download] I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply] [d/l]
Re: How to dump a PDL by Anonymous Monk on Dec 26, 2010 at 02:22 UTC
I don't know if it would affect execution time -- it might, by avoiding `push` -- but a Perlish way to do that would be: `my @scalar = map { $piddle->at($_) } 0 .. ($piddle->getdim(0) - 1);` I'm not familiar at all with PDL, but a quick look at the core docs shows that you can do this: `my @values = $piddle->at(0 .. ($piddle->getdim(0) - 1));` so I would assume there's an even easier way to do what you want in a single statement somewhere (maybe `list`, but the docs indicate that it's a last resort function).	[reply] [d/l] [select]
Re: How to dump a PDL by etj (Priest) on May 28, 2022 at 21:34 UTC
Since this was originally posted, PDL acquired an `unpdl` method/function which can be used to turn an ndarray into a Perl array-ref structured like the ndarray. That would end up with a similar result to the posted code, but with a lot less performance-draining jumping between C and Perl. Also, I believe a better approach is to use PDL::IO::CSV or PDL::IO::Misc#wcol to store CSV data, not YAML.	[reply] [d/l]