Re: Parallel::Forkmanager and large hash, running out of memory

If I understand the crux of the matter is: You need to look up data in a file that is too big to hold in RAM?

Perhaps you can build an index for the file that is smaller and then use that to look into the file. If performance is an issue memoizing your lookup may help. But then again premature optimization and all that...

You may also want to put it into a DB. SQLite is great way if you just want to create it then use it for lookups. Not so great if you need concurrent access, then you may need a real DB server

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Comment on Re: Parallel::Forkmanager and large hash, running out of memory

Replies are listed 'Best First'.
Re^2: Parallel::Forkmanager and large hash, running out of memory by mabossert (Scribe) on Apr 24, 2013 at 15:25 UTC
Gotcha...thanks for the quick response. Will SQLite handle large sizes? I thought it had a 2GB limit...or am I smoking proverbial crack?	[reply]
Re^3: Parallel::Forkmanager and large hash, running out of memory by Anonymous Monk on Apr 24, 2013 at 15:30 UTC
https://sqlite.org/limits.html	[reply]
Re^4: Parallel::Forkmanager and large hash, running out of memory by mabossert (Scribe) on Apr 24, 2013 at 16:42 UTC
Got it...I guess I need to lay off the crack pipe...Thanks! Of course, this brings me to the next challenge: concurrency. Since I will be doing this work in parallel, I will have to see how SQLite handles concurrency...or perhaps lean over towards MySQL... Thanks again for the help to you (and all others that responded...)	[reply]
Re^5: Parallel::Forkmanager and large hash, running out of memory by pokki (Monk) on Apr 24, 2013 at 18:37 UTC