in reply to Enquiry on memory usage

I would like to know is this script able to run on the server without overloading?

Yes, easily. Your data will require ~1.8GB of memory.

Or is there any other better way of doing this without putting much load on the memory?

Possibly. You could sort all 3 files and read them in parallel, and output the information line by line. This would use minimal memory and isn't difficult to program, though there are a lot of edge cases that are easy to get wrong.

But given that your current solution will use less than 3% of the server's capacity, or 3.5% of your maximum allocation, there seems no reason to move away from the simple direct approach.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^2: Enquiry on memory usage
by faozhi (Acolyte) on May 08, 2009 at 08:09 UTC
    Cheers for that. Can I know how you calculate it?
[reply]

      Sure. I ran the following command which creates a 1 million key HoHoH, and then pauses: 

      perl -e"$h{ $_ }{1} = { 1 .. 4 } for 1..1e6; <>"

      [download]

      I then looked at my process monitor and saw that it required 600MB of memory, so I multiplied by 3 (the number of files) to come up with ~1.8GB. I've made a lot of assumptions. eg. that the chromosomes within the 3 files substantially overlap. I could have multiplied by 5 and arrived at 3GB for example. My guess is that your actual requirement will be somewhere between.

      But either way, the total memory requirement is very unlikely to seriously threaten your 50GB maximum, so there was no need to suggest that you change your methods.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        Cheers mate. That cleared things up. :)
[reply]

In Section Seekers of Perl Wisdom