in reply to Force perl to release memory back to the operating system

Oh brothers! Thank you very much everyone for offering your great advices!

I took the advise and applied some very basic assumptions to reduce the data set first.

My new assumption is that top 30 customers will most likely have balances greater than 0.5 billion dollars. So as a first step, I added the following code to reduce my data set, having previously obtained a hash table of customers & balances.
while (...) { $balances{$cust} += $bal * $fx_rates->{$ccy}; } ... # Find the customers whose total balance is greater than $500000000. +00 my %top_balances; my $top_threshold = 510000000.00; # default threshold $0.5 bil my $top_count; # keep a count of how many found do { # lower threshold by 10 million every time $top_threshold -= 10000000.00; die ("This is impossible, the data has error!") if ($top_threshold < 0.00) $top_count = 0; while ( my ($cust, $balance) = each %balances ) { if ($balance > $top_threshold) { $top_balances{$cust} = $balance; $top_count++; } } } while ($top_count < $nth); # Safety check - how big is our list? Is it > n? if ($VERBOSE) { print "$top_count customers have ", "balances greater than \$${top_threshold}.\n"; } # Sort only the top customers by descending order of balance my @sorted_cust_list = reverse map { $_->[0] } sort { $a->[1] <=> $b->[1] } map { [$_, $top_balances{$_}] } keys %top_balances;
As it turned out, the %balance structure occupied less than 400Mb in memory. It was the Schwartzian transform on the entire %balance table that thrashed the machine.

When I ran the new code, it showed that the reduced set %top_balances only has less than 100 customers. So I was able to keep the Schwartzian transform, knowning that I only have to sort about 100 customers in the reduced customer set.

As it turned out, I don't have to fork/restart/reclaim perl memory at all. Also the modified script was able to finish everything (including the extraction of top 30 customer details) under 6 minutes! In comparison it used to run more than 20 minutes and was killing the machine.

A big THANK YOU to everyone for the great help!

Replies are listed 'Best First'.
Re: Re: Force perl to release memory back to the operating system
by blssu (Pilgrim) on Sep 26, 2003 at 14:06 UTC
    You have accounts receivable of 6,000,000,000 dollars and you're not using a database to store transactions? Which cartel do you work for?