in reply to Counter - number of tags per interval

50 distinct id's and each id in file.2 has approximately 5-7 * 10^9 lines.

File 2 has ~350 billion lines?

What is the maximum coordinate?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: Counter - number of tags per interval

Replies are listed 'Best First'.
Re^2: Counter - number of tags per interval
by baxy77bax (Deacon) on May 04, 2013 at 15:23 UTC
    yes that is true (why is this so odd ?? in bioinformatics you have 50-500gb read-mapps (files) and nobody finds that odd :) ). Maximum coordinate ?? well that depends on a simulation. but i would say for id = a, max is app. 5*10^9 i mean i need to find this out by going through the file.
[reply]
      why is this so odd ?? i

      Not odd. Just important to know as it greatly affects the possible solutions.

      How long does/do you estimate it to take to process those two files with your current code?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        current estimates:

        One Id app. 8-12 h
        50 id's app. 500h = app. 20 days which is 20 times longer then the actual computation
        since there will be more of such simulations i think a better solution would be preferable. Unfortunately i cannot temper with the simulation part since i have no source , but even if i did i wouldn't do it since there are a lot of heuristics involved which i honestly don't understand.

[reply]

In Section Seekers of Perl Wisdom