Re: memory:: grep vs looping through hash

Premature optimization is the root of all evil.
— Donald E. Knuth

As for your memory concern: the best thing to do is just try it. In almost all cases you'll find it won't be a problem.
I made it a bit shorter for you, though:

# untested
%inter = 
    map { $_ => ($hist1{$_} <= $hist2{$_} ? $hist1{$_} : $hist2{$_}) }
    grep { $hist2{$_} } keys %hist1;
[download]

Should you run into problems, it's probably less of a headache to throw more memory at it. Purchase it for your PC or server, or rent from one of the popular "cloud" providers that charge only several cents an hour for quite a bit of memory.

-Thomas

"Excuse me for butting in, but I'm interrupt-driven..."

Comment on Re: memory:: grep vs looping through hash Download Code

Replies are listed 'Best First'.
Re^2: memory:: grep vs looping through hash by f77coder (Beadle) on Aug 18, 2014 at 07:21 UTC
Thomas, thanks for reply/code. My system is already maxed and have already hit the wall with programs R and Matlab so I need something brutally fast at strings/IO. In your opinion/experience the grep will be faster?	[reply]
Re^3: memory:: grep vs looping through hash by thomas895 (Deacon) on Aug 19, 2014 at 01:09 UTC
My system is already maxed and have already hit the wall with programs R and Matlab so I need something brutally fast at strings/IO In that case, try my suggestion of renting a server somewhere to do this for you. Digital Ocean is a popular one, and their prices range from USD 0.007/hour for 512MB of RAM to USD 0.952/hr for 64GB with a 20-core processor. One of their big selling points is that when you're not using it, you simply "turn it off" and don't pay anything. You get a virtualised Linux instance to do whatever you'd like with. Amazon also offers their "EC2" service which is essentially the same thing -- but beware of large corporations trying to lock you in! HP has one as well, I believe, as does Microsoft. In your opinion/experience the grep will be faster? I really couldn't say. The most largest array I've grep'd was maybe a few hundred members. -Thomas "Excuse me for butting in, but I'm interrupt-driven..."	[reply]