in reply to Consolidating nstore arrays

This is not turn key code, but hopefully it will provide some guidance.
Some problems:
I don't understand your pattern, it doesn't look perlish.
Certainly comment out test data generation.
There will be problems if input files have different sizes.
This uses my current favorite Path::Tiny but could be adopted to others. 

#!/usr/bin/perl

# http://perlmonks.org/?node_id=1198355

use strict;
use warnings;
use Path::Tiny;
use Storable qw(nstore retrieve);

my $column = 1;                              # these could be passed i
+n...
my $filepattern = qr/(thing.*).foo__bar_ar/;
my $projectdirectory = './some';

# make test data

path("$projectdirectory/$_")->mkpath for qw( one two/three );
nstore [[1.34, 2.53], [3.26, 4.001]], "$projectdirectory/one/thing1.fo
+o__bar_ar";
nstore [[5.55, 6.911], [7.373, 8.808]],
  "$projectdirectory/two/three/thing3.foo__bar_ar";

# end make test data

my @answer;
my $header = 'Number';

path($projectdirectory)->visit( sub
  {
  my ($path) = @_;
  /$filepattern/ or return;
  $header .= ", $1";
  my $arrayref = retrieve($path);
  my $number = 0;
  $answer[$number++] .= ", " . $_->[$column] for @$arrayref;
  },
  { recurse => 1 }
  );

my $number = 1;
$_ = $number++ . "$_\n" for @answer;

print "$header\n", @answer;

[download]

Replies are listed 'Best First'.
Re^2: Consolidating nstore arrays
by Speed_Freak (Sexton) on Aug 31, 2017 at 19:50 UTC

    I've been able to swap in my project directory and get this to partially work. I get 12x error "Use of uninitialized value in concatenation (.) or string at test.pl line 35." And line 35 corresponds to this line:

     $answer[$number++] .= ", " . $_->[$column] for @$arrayref;

    I am searching the directory for 12 files, so that seems to make sense that there are 12 errors. Once the errors print,I get the header row, and then I get the number 1 row as this: 1, , , , , , , , , , , ,. Then all of the following rows print just like intended.

    When I try to point to the column I want...(in this case, the 11th column. $column = 11) I get 3,635 of those errors before getting the blank row 1, and then all of the data that I want.

     nohup:
  Use of uninitialized value in concatenation (.) or string at test.pl
+ line 35.
Number, thing1, thing2, thing3, etc
1, , , , , , , , , , , , 
2, 0.954370854376634, 0.342822118341448, 0.572790744083124, 0.59224361
+0847652, 0.8851590415068, 0.991382122632363, 0.507483754645392, 0.911
+63751726292, 0.746774421369453, 0.834212591847216, 0.405652973225827,
+ 0.816633320526871
3, 0.991624960933286, 0.991327116256435, 0.941102605547385, 0.99452435
+3181868, 0.994089351524488, 0.936750116018622, 0.984664588090976, 0.9
+91809613768339, 0.995157524403383, 0.99036040081599, 0.5066156863869,
+ 0.559803287167354
4, 0.971949973836183, 0.98269241861483, 0.98789028053774, 0.9698163919
+63118, 0.967753189092842, 0.951648687904388, 0.993352530560803, 0.988
+214472795065, 0.989701083946332, 0.982533372295779, 0.271015752055197
+, 0.563841526116344
5, 0.035742211264196, 0.365226753494403, 0.2865774134715, 0.2099523224
+1765, 0.472126907503153, 0.380517314225538, 0.281019146329213, 0.2162
+24917348467, 0.0665524840406343, 0.175285452052, 0.4819820327753, 0.4
+23563619759855
6, 0.756579058259939, 0.890650837623651, 0.931449842413731, 0.76140477
+0602313, 0.704484524751191, 0.399501625385289, 0.516747548785127, 0.8
+25989268175438, 0.590967055354945, 0.524160278838733, 0.1661424041710
+08, 0.000491664086159459
7, 0.712831728096357, 0.254713361402047, 0.539928335806198, 0.08034247
+23962962, 0.0344976765160731, 0.182113998255013, 0.14620549377983, 0.
+0195129814144615, 0.532640937604125, 0.283745467826306, 0.34889286275
+5017, 0.483644189994046

    [download]

    One other problem I am having, I need the pattern to read more like my $filepattern = qr/(*).foo__bar_ar/ because the file names can change substantially, but the extension is always the same. But I get this error "Quantifier follows nothing in regex; marked by <-- HERE in m/(* <-- HERE ).foo__bar_ar/ at test.pl line 11." when I try things like * or *.*

    I've posted a small set of data sets in a reply to the main thread.

[reply]
[d/l]
[select]

      You are missing a . in your pattern: 

      my $filepattern = qr/(.*).foo__bar_ar/

      [download]

      Also, I'm totally confused about what is in your retrieve files.
      Please show (in code blocks) at least the first twenty lines from a Data::Dumper or Data::Dump print of one of your "retrieve" files as it is immediately after it is read in.

[reply]
[d/l]

        ah, thanks! Below is a dump from one file. 

        $VAR1 = undef;
$VAR2 = [
          1,
          '18.4',
          '7.6',
          '10.8',
          '0.584615384615385',
          22,
          '4.0',
          18,
          '0.307692307692308',
          '0.664861632672521',
          '0.968405008381221',
          '0.816633320526871'
        ];
$VAR3 = [
          0,
          '31.5',
          '18.9',
          '12.6',
          '0.75',
          '199.7',
          '29.2',
          '170.5',
          '0.255133245958934',
          '0.150796748317197',
          '0.968809826017511',
          '0.559803287167354'
        ];
$VAR4 = [
          0,
          '115.2',
          '35.9',
          '79.3',
          '0.475181998676373',
          '13.7',
          '8.3',
          '5.4',
          '0.754545454545455',
          '0.855054749249092',
          '0.272628302983597',
          '0.563841526116344'
        ];
$VAR5 = [
          0,
          '969.7',
          '1034.6',
          '-64.8999999999999',
          '1.03238038217832',
          '1607.6',
          '582.0',
          '1025.6',
          '0.531603945926197',
          '0.0340815410482703',
          '0.81304569847144',
          '0.423563619759855'
        ];
$VAR6 = [
          0,
          '3.2',
          '13.2',
          -10,
          '1.60975609756098',
          '22.2',
          '58.2',
          -36,
          '1.44776119402985',
          '0.000189855866797165',
          '0.000793472305521753',
          '0.000491664086159459'
        ];
$VAR7 = [
          0,
          124,
          '24.9',
          '99.1',
          '0.334452652787105',
          '533.5',
          '764.2',
          '-230.7',
          '1.17777606534638',
          '0.959457725728336',
          '0.00783065425975528',
          '0.483644189994046'
        ];

        [download]

        and so on for ~150k VAR's

[reply]
[d/l]

In Section Seekers of Perl Wisdom