Re^4: Reducing memory footprint when doing a lookup of millions of coordinates

Cool. That makes sense. Cheers

There are 24 chromosomes I'm interested in in the file with the total number of records (ranges) per chromosome as follows:

chr1 2235512
chr2 674652
chr3 348269
chr4 323500
chr5 308100
chr6 338158
chr7 280734
chr8 253229
chr9 224412
chr10 237524
chr11 240186
chr12 250300
chr13 161894
chr14 160126
chr15 152561
chr16 170145
chr17 167623
chr18 126566
chr19 134123
chr20 123693
chr21 61077
chr22 75260
chrX 265561
chrY 43169
[download]

The length of the ranges are usually between about 10-300.

Does this help?

Comment on Re^4: Reducing memory footprint when doing a lookup of millions of coordinates Download Code

Replies are listed 'Best First'.
Re^5: Reducing memory footprint when doing a lookup of millions of coordinates by BrowserUk (Patriarch) on Feb 27, 2011 at 13:40 UTC
Does this help? No. But from those figures I have to agree with moritz that splitting the dataset into 24 files and loading each set individually makes perfect sense. It would hardly affect your performance at all, but reduce your memory consumption to the size of the largest set. Ie. roughly 2235512/7356374 = 30% of 1.2GB ~= 350MB. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]

Replies are listed 'Best First'.

Re^5: Reducing memory footprint when doing a lookup of millions of coordinates
by BrowserUk (Patriarch) on Feb 27, 2011 at 13:40 UTC

Does this help?

No.

But from those figures I have to agree with moritz that splitting the dataset into 24 files and loading each set individually makes perfect sense. It would hardly affect your performance at all, but reduce your memory consumption to the size of the largest set.

Ie. roughly 2235512/7356374 = 30% of 1.2GB ~= 350MB.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]