in reply to load a file in memory and extract parts

Obviously, the first notion that pops into my mind is:   “this would be a great application for an SQLite database file!”

SQLite, as you know, is a powerful SQL system that stores everything into just one file.   It already has existing facilities to import a CSV-file.   It can quite-effortlessly import your file into a table, then create indexes on that table to allow the key (which consists of two columns ...) to be retrieved very fast.   You can, if you like, even import the incoming file into another table, and then your entire problem has basically been reduced to an INNER JOIN.

Food for thought, anyway.   Personally, I love to stumble-upon situations where I don’t have to write a single line of “programming” in order to solve a problem.   Maybe this will work for you; maybe not.   But if it does, you could chomp this entire problem in a matter of minutes.

Replies are listed 'Best First'.
Re^2: load a file in memory and extract parts
by healingtao (Novice) on May 06, 2015 at 16:36 UTC

    I tried the following code 

    sub TestArray {
my @file_array;
my $InputFile = "/home/test/test_file.dat"; # '|' separator file
my $column;
my $row;

tie @file_array, 'Tie::File', $InputFile or die $!;
shift @file_array;

foreach $row (0..@file_array-1)
{
        foreach $column (0..@{$file_array[$row]}-1)
        {
            $LOG->info( "Element [$row][$column] = $file_array[$row][$
+column]\n");
        }
}
}

    [download]

    At this point I'm trying to load the file into a multidimentional array. The file has 10 columns and 50,000 rows. How can I extract data from this array. When I run this, I'm getting the following error: Can't use string ("DEAL_NAME|TRANCHE_NAME|GROUP_NO|") as an ARRAY ref while "strict refs" in use at ./test_feeds.pl

[reply]
[d/l]
      Tie::File lets you access each line of your file as an array element. It does not load the file into a multidimensional array. The reason for the warning is that you are trying to access an element in an Array-of-Arrays data structure, but you just have a simple array (perldsc). Maybe split will help: 
      use warnings;
use strict;

my @file_array = qw(a|b|c d|e|f);
foreach my $row (0 .. @file_array - 1) {
    foreach my $column (split /\|/, $file_array[$row]) {
        print "$row $column\n";
    }
}

__END__

0 a
0 b
0 c
1 d
1 e
1 f

      [download]
[reply]
[d/l]

        Actually that didn't work, I get the same error at line foreach my $row (0..@file_array-1) Can you try it with tie by loading the following file col1|col2|col3|col4|col5 data1|data2|data3|data4|data5 data1b|data2b|data3b|data4b|data5b Thanks

[reply]

In Section Seekers of Perl Wisdom