in reply to Re: load a file in memory and extract parts
in thread load a file in memory and extract parts

I tried the following code 

sub TestArray {
my @file_array;
my $InputFile = "/home/test/test_file.dat"; # '|' separator file
my $column;
my $row;

tie @file_array, 'Tie::File', $InputFile or die $!;
shift @file_array;

foreach $row (0..@file_array-1)
{
        foreach $column (0..@{$file_array[$row]}-1)
        {
            $LOG->info( "Element [$row][$column] = $file_array[$row][$
+column]\n");
        }
}
}

[download]

At this point I'm trying to load the file into a multidimentional array. The file has 10 columns and 50,000 rows. How can I extract data from this array. When I run this, I'm getting the following error: Can't use string ("DEAL_NAME|TRANCHE_NAME|GROUP_NO|") as an ARRAY ref while "strict refs" in use at ./test_feeds.pl

Replies are listed 'Best First'.
Re^3: load a file in memory and extract parts
by toolic (Bishop) on May 06, 2015 at 16:52 UTC
    Tie::File lets you access each line of your file as an array element. It does not load the file into a multidimensional array. The reason for the warning is that you are trying to access an element in an Array-of-Arrays data structure, but you just have a simple array (perldsc). Maybe split will help: 
    use warnings;
use strict;

my @file_array = qw(a|b|c d|e|f);
foreach my $row (0 .. @file_array - 1) {
    foreach my $column (split /\|/, $file_array[$row]) {
        print "$row $column\n";
    }
}

__END__

0 a
0 b
0 c
1 d
1 e
1 f

    [download]
[reply]
[d/l]

      Actually that didn't work, I get the same error at line foreach my $row (0..@file_array-1) Can you try it with tie by loading the following file col1|col2|col3|col4|col5 data1|data2|data3|data4|data5 data1b|data2b|data3b|data4b|data5b Thanks

[reply]

        If you want to learn by experimenting, remember that a string and an array are very different data structures. Tie::File gives you an array of strings, but you treat it like an array of arrays (containing strings). Perl constructs arrays of arrays as arrays containing array references. See perldsc. Use Data::Dumper to see what your data actually looks like.

        If you just want to solve the original problem, don't parse CSV manually, use Text::CSV_XS to get your data in a format that you can work with. Consider using DBI and DBD::CSV instead for an even shorter way (the SQL INNER JOIN proposed by locked_user sundialsvc4) to read AND combine AND export your data. Note that the DBI way also allows to migrate the program to SQLite or any other database in a later version.

        Unless you know your CSV files in every single detail, and especially unless you are absolutely sure that they don't contain nasty things like optional quotes, quoted separator chars, escaped quotes, newlines in data, NULLs, binary data, and so on, you better AVOID split and use Text::CSV_XS. split alone can't handle any of those problems, Text::CSV_XS can handle all of them.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom