4**21 = 4,398,046,511,104. If each of those array elements took just 1 byte that would need 4000GB (or 4TB) of ram.

But each element of a Perl array requires a minimum of 32bytes on a 64-bit system, so that increases the programs memory requirement to at least 32,000GB.

And that's before you multiply by 197!

Even today, very few computers can access Terabytes of memory -- the latest greatest SkyLake will only address 128GB.

The bottom line is that you are going to have to tackle your problem a different way.

If you can describe what you are trying to do -- in computer terms, with a minimum of bio-lingo -- there is probably an approach that will allow you to achieve it without needing £432,000 worth of memory and a processor that can address it; if such an animal currently exists, it would be in the $millions price range.

Hi! Thank you so much for your reply! So the goal I try to achieve here is that as for the 21mer I'm working on, it is a sequence of 21 letters, each position can have 4 variable "AGCT". Therefore, in total, it should have 4**21 combination. I have total 197 kmer files, each of them contains around 8000 of these 21mers, and I'm trying to count how many time each 21mers has shown among these 197 files. That's why I need a matrix to do so.

Therefore, in total, it should have 4**21 combination.

Yes. But not all of them -- in fact a very small percentage -- will appear; so rather than reserving space for trillions of counts that will 99% never be used, you can choose to only count those that do appear. That will save huge amounts of space and speed up your processing.

With 197 files * 8000 lines, there are at most 1.5 million 21kmers -- if they are all unique -- so there in no need to reserve space for all 4.3 trillion possibilities. 197*8000 / 4.3e12 * 100 = 0.0000358%

I'm trying to count how many time each 21mers has shown among these 197 files.That's why I need a matrix to do so.

Fair enough, but you don't need to count them all in one huge matrix. You can process the 197 files one by one and only store the nonzero counts across files.

Could you post a small sample (say 20 lines) of what is in your kmer files?

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.

In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit

197 * 8000 = 1576000. If you only have 1.5e6 data points why are you trying to match them against 4**21 possible combinations when all bar an infinitessimal fraction of those won't be present?

Just compare your 1.5e6 against themselves and mark everything else as zero. No matrices required.

For fun, see The 10**21 Problem (Part I) and the subsequent parts.

(Edit: BrowserUk managed to construct a fuller and therefore better response in the time it took me to deduce and type this. Still, great minds and all that ;-)

Hello!

It's hard to see whole picture without sample of data, but i will try.

As far as i can see, you have files with data something like

1 the only
2 significant
3 part
4 is 
5 a 
6 number
7 before 
8 the 
9 first 
10 space 
11 character
[download]

Then you pick the leading number in every string and use it as an index of array @sum to sum the number of occurrences of this number. I don't see why you even need the @matrix array at this point, maybe you will use it later. Will you really need 4**21 size of an array? Will there be 4**21 different numbers in those files? And if not, if your memory actually can contain all numbers in your files (but not the range of these numbers), then maybe you should use a hash instead of an array? Something like this:

#!/usr/bin/perl
use strict;
use warnings;

my %matrix;
my %sum;

#inport file names
my $fn = $ARGV[0];
chomp $fn;
open (my $FILE, $fn) or die "Can't open file \"$fn\"\n\n";

chomp(my @line = <$FILE>);
my $n;

for ($n=0; $n<scalar @line; $n++){
#open each kmer file
    open(my $KMER, $line[$n]) or die "Can't open file \"$line[$n]\"\n\
+n";
    my @kmers = <$KMER>;
    my @all_kmer;
    my $kmer;
    foreach $kmer (@kmers){
        my @split = split(/\s+/,$kmer); #obtain kmer reference number
        # for now we don't have to populate the big %matrix, commented
+ 
        # $matrix{$n}{split[0]} = 1;
        $sum{$split[0]}++;
    }
    close $KMER;
}
close $FILE;

#creat outfile
my $outfile = $ARGV[1];
open (my $OUT, '>' .$outfile) or die "\nUnable to create $outfile\n";

#sum up
foreach my $k (keys %sum){
    print $OUT "$sum{$k}\n";
}
[download]

Still, it's hard to tell will it suffice without the data. And if not, and count of numbers will not fit in your RAM, then you will have to use the interim DB, like MySQL or similar. I will use MySQL queries. First, create DB and table in it:

CREATE DATABASE kmers;
CREATE TABLE matrix (file int, number int, key (file), key (number));
[download]

Then you will have to populate matrix table with SQL query like this:

use DBI; 
my $dbh = DBI->new('DBI:mysql:database=kmers;host=127.0.0.1','mysql_us
+er','');
my $query = "INSERT INTO matrix VALUES (?,?)";
my $sth = $dbh->prepare($query);

for ($n=0; $n<scalar @line; $n++){
#open each kmer file
    open(my $KMER, $line[$n]) or die "Can't open file \"$line[$n]\"\n\
+n";
    my @kmers = <$KMER>;
    my @all_kmer;
    my $kmer;
    foreach $kmer (@kmers){
        my @split = split(/\s+/,$kmer); #obtain kmer reference number
        # for now we don't have to populate the big %matrix, commented
+ 
        # $matrix{$n}{split[0]} = 1;
        # $sum{$split[0]}++;
                $sth->execute($n,$split[0]);
    }
    close $KMER;
}
$sth->finish();
[download]

And after matrix table populated you can query sum from it

my $count_query = "SELECT number,COUNT(number) FROM matrix GROUP BY nu
+mber ORDER BY number";
$sth = $dbh->prepare($count_query);
[download]

But still, even with interim DB, if the count of numbers is greater than your memory, you will have to fetch and print result to file by one row at a time:

#sum up
#foreach my $k (keys %sum){
#    print $OUT "$sum{$k}\n";
#}

$sth->execute(); 
while (my $res = $sth->fetchrow_arrayref()) {
   print $OUT $res->[1]\n";
}
[download]

Hi! the hash actually resolves my problem!! I was thinking too complicated but what I actually needed is just the reference number and how many time it has appeared!! Thank you so much!

This node is a properly HTML-tagged version of your previous Reaped: Out of Memory when generating large matrix node, which used incorrect  <code> ... </code> tags. For future reference, it would have been better if you had fixed the previous node rather than posting what looks like an exact duplicate except for proper tags. Please see How do I change/delete my post? (Please do not fix the previous node now as I have already considered it for removal and, if other monks agree, it will, in the fullness of time, disappear.) Please also see Markup in the Monastery and Writeup Formatting Tips.

If your matrix contains many zeros, you may wish to consider a Sparse Matrix.