Re: Re: Force perl to release memory back to the operating system

The first part of the collection process -

$balances{$cust} += $bal * $fx_rates->{$ccy};
[download]

calculates the total balance for each customer, where each customer can have transactions in multiple currencies. I have thought about applying certain heuristic, unfortunately I can't make assumptions about customer behaviours (not to mention the consequence of producing an approximate report).

I am stuck with storing balances in memory. I don't want to export these balances into temporary files, because that will introduce significant penalties on storing 10+ million individual accounts to disk and reading them back in again. I will get rid of the memory hungary Schwartzian transform and use a sorting technique that will be balanced on memory usage and speed.

At the end of the selection process, I will have a list of top 30 customers. Come to think about it, I will probably join the 30 customer info in a string of comma separated values, and restart the perl script again with the exec { $^X, $0, @ARGV } technique.

I will go to sleep tonight with these ideas in my mind, I will probably get enlightened in my dreams. :-)

And thanks again for your suggestions.

Comment on Re: Re: Force perl to release memory back to the operating system Select or Download Code

Replies are listed 'Best First'.
Re: Re: Re: Force perl to release memory back to the operating system by iburrell (Chaplain) on Sep 25, 2003 at 21:18 UTC
Have you thought about using a tied hash? DB_File would work perfectly since you are using a large hash that is bigger than RAM. The Berkeley DB library handles all the nasty details of saving the data that won't fit in memory. In addition, DB_File supports in-memory databases that back to disk so you wouldn't even have to worry about creating and deleting a temp file. Since you are selecting a limited number of the highest values, keeping a list of the highest values see so far while scanning through the entire hash will be much more efficient than operating on everything at once.	[reply]

Replies are listed 'Best First'.

Re: Re: Re: Force perl to release memory back to the operating system
by iburrell (Chaplain) on Sep 25, 2003 at 21:18 UTC

Since you are selecting a limited number of the highest values, keeping a list of the highest values see so far while scanning through the entire hash will be much more efficient than operating on everything at once.

[reply]