in reply to Force perl to release memory back to the operating system
is central to what you're doing. Unless there's some heuristic you can apply to exclude some customers (i.e., "forget these, since they'll never be anywhere near the top N"), you may be stuck here. How many customers do you have?$balances{$cust} += $bal * $fx_rates->{$ccy};
Moving on,
is the second hit, and it's a big one. First, keys %balances builds a big array, then you build another one via sort, then another via reverse. (Not all of the arrays are live all the way through the pipeline, but still...) There's a way around this, though. Instead of using a Schwartzian transform, make a low-footprint pass through the hash, usingmy @sorted_cust_list = reverse map {$_->[0]} sort {$a->[1]<=>$b->[1]} map{[$_,$balances{$_}]}keys %balances;
and keep a smaller data structure that records the top N customers by balance. (I'll leave the choice of algorithm as an exercise for the motivated reader.) This is a bit more work, but it might run faster given the reduction in memory demands and attendant reduction in swapping/thrashing.while ( my($cust,$balance) = each %balances ) { ... }
At this point, the big memory sink, assuming you have lots of customers, is %balances. You can undef this at the end of the subroutine, but, as far as I know, this won't result in Perl releasing memory back to the OS (and it goes away at the end of the routine anyway). The last time I looked into it, which was a few releases of Perl ago, Perl indirectly used a "raise the watermark if necessary" memory allocation strategy, with no provision for ever lowering the watermark.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: Re: Force perl to release memory back to the operating system
by Roger (Parson) on Sep 25, 2003 at 12:35 UTC | |
by iburrell (Chaplain) on Sep 25, 2003 at 21:18 UTC |