in reply to Re^3: Out of Memory when generating large matrix
in thread Out of Memory when generating large matrix
IF the RAM wasn't even sufficient for counting, presorting a giant file wouldn't help. (wc could help but an output with 1.5 million colons should be avoided...)
I'd surely opt for a DB like SQLite.
But I suppose only some thousands of the most frequent K-mers are of interest.
(I can imagine a solution with Hash Of Hashes for counting. Only the most relevant hashes are kept in memory while the others are swapped out, but I this would extent the scope of this thread.)
Cheers Rolf
(addicted to the Perl Programming Language and ☆☆☆☆ :)
Wikisyntax for the Monastery
|
---|