in reply to Re: Re: Force perl to release memory back to the operating system
in thread Force perl to release memory back to the operating system

It might be time to rethink that business decision. Anyway you can save some memory if you think about what you are doing.

# get the customer => blance hash mapping - if you can't RDBMS you are + stuck with this $cust_bal = ...... # now all we are interested in is the top 30 # we don't need to sort them all using a Schwartzian # we only need to sort the values. This saves us a lot of memory as # [ cust, bal ] will take up a lot more space than just bal # we can also exit at this point and write a new temp flat file # with all the cust => bal mappings and this reclaims the memory. # either way we just want those top 30 so just sort the balances: my @bals = sort { $b <=> $a } values %$cust_bal # we are only interested in the top 30 which is a balance greater than +.... my $gt = $bal[30] # so now we iterate over our hash again (or the flat file) my $top_30; for my $cust( keys %$cust_bal ) { next unless $cust_bal > $gt; $top_30->{$cust} = $cust_bal->{$cust}; } # now the top_30 are in a hash ready to be sorted for mapping to cust +details and output

This will save you roughly 25% and up to 50% (by using a temp file) of the memory that the original algorithm used which might get you home. If not you will have to tie your hash to a file. dbmopen might be the DB you have when you are not having a DB :-) If you can't do it in memory and have to tie to the disk it will be really slow, but I gues you have got all day :-)

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

  • Comment on Re: Re: Re: Force perl to release memory back to the operating system
  • Download Code

Replies are listed 'Best First'.
Re: Re: Re: Re: Force perl to release memory back to the operating system
by Roger (Parson) on Sep 25, 2003 at 11:56 UTC
    Hi brother Tachyon, thanks for your suggestion! It has openned up my mind. I have never thought about the memory savings by using a straight sort. Too bad I am back home already, I can't wait to get back to office tomorrow morning to try it out.

    Thanks again for your prompt reply! I really appreciate it. I will let you how I go tomorrow!
Re: Re: Re: Re: Force perl to release memory back to the operating system
by Jasper (Chaplain) on Sep 25, 2003 at 10:43 UTC
    I don't think this is solving anything, because (as I understand the OP) the files are of transactions, not customers and balances.

    If each entry in the file was a customer and a balance, one run through would be sufficient, just keeping the top 30 customers.

    It's more complicated than that, and a nightmare without a RDB.

      You can that there are multiple transactions per customer with the bit of code that has:

      $hash_ref->{$cust} += balance * $fudge_factor

      Agree PHB needs shooting. Saying you can't use a RDBMS when it is suited to the job is like saying OK David, I'm sorry but you can't use one of the surplus RPGs we are tripping over, but can you go an kill that really big Goliath guy over there with this slingshot..... Oh and we are out of pebbles too. But don't worry I have faith that you will get this done on time and under budget.....

      cheers

      tachyon

      s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print