Oh brothers! Thank you very much everyone for offering your great advices!

I took the advise and applied some very basic assumptions to reduce the data set first.

My new assumption is that top 30 customers will most likely have balances greater than 0.5 billion dollars. So as a first step, I added the following code to reduce my data set, having previously obtained a hash table of customers & balances.
  while (...)
  {
    $balances{$cust} += $bal * $fx_rates->{$ccy};
  }
  ...
  # Find the customers whose total balance is greater than $500000000.
+00
  my %top_balances;
  my $top_threshold = 510000000.00;   # default threshold $0.5 bil
  my $top_count; # keep a count of how many found

  do {
    # lower threshold by 10 million every time
    $top_threshold -= 10000000.00;
    die ("This is impossible, the data has error!")
        if ($top_threshold < 0.00)
    $top_count = 0;
    while ( my ($cust, $balance) = each %balances )
    {
      if ($balance > $top_threshold)
      {
        $top_balances{$cust} = $balance;
        $top_count++;
      }
    }
  } while ($top_count < $nth);
  # Safety check - how big is our list? Is it > n?

  if ($VERBOSE) {
    print "$top_count customers have ",
          "balances greater than  \$${top_threshold}.\n";
  }

  # Sort only the top customers by descending order of balance
  my @sorted_cust_list =
    reverse map { $_->[0] }
    sort { $a->[1] <=> $b->[1] }
    map { [$_, $top_balances{$_}] }
    keys %top_balances;

[download]
As it turned out, the %balance structure occupied less than 400Mb in memory. It was the Schwartzian transform on the entire %balance table that thrashed the machine.

When I ran the new code, it showed that the reduced set %top_balances only has less than 100 customers. So I was able to keep the Schwartzian transform, knowning that I only have to sort about 100 customers in the reduced customer set.

As it turned out, I don't have to fork/restart/reclaim perl memory at all. Also the modified script was able to finish everything (including the extraction of top 30 customer details) under 6 minutes! In comparison it used to run more than 20 minutes and was killing the machine.

A big THANK YOU to everyone for the great help!

In reply to Re: Force perl to release memory back to the operating system by Roger
in thread Force perl to release memory back to the operating system by Roger

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.