------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

I am processing them one by one, it appears that perl keeps chewing space anyways. Here's the relevant section of code:

    foreach (sort keys %tickets)
    {
    my $ticket = ((split " ",$_)[0]);
    my %tick_inf;                            #hash to hold ticket info
    (%tick_inf = ars_GetEntry($rem,$schema,$ticket))
        || warn "Could not retrieve $ticket: $ars_errstr";    #populat
+e the hash with the ticket info
    my $field_data=$ticket;
    foreach (@$afield_names){    #these are the fields that we have be
+en asked to grab.
        $field_data .= " " . $sep;                #join the previous d
+ata with the seperator 
        unless (exists $hfield_names->{$_})
        {
            print "Bad field name -- $_\n";
            next;
        }
        unless (exists $tick_inf{$hfield_names->{$_}})
        {
            print $hfield_names->{$_} . "\n";
            print "Bad ticket info identifier -- $_ does not relate to
+: " . $hfield_names->{$_} . "\n";
            next;
        }
        unless ( defined($tick_inf{$hfield_names->{$_}}))
        {
            $field_data .= " ";
            next;
        }
        if (ref($tick_inf{ $hfield_names->{$_} }) eq "ARRAY")    #For 
+array references
        {
            my $data;
            foreach (@{ $tick_inf{ $hfield_names->{$_} } }) #foreach o
+f the values of the array ref
            {
                $data .= expand_hash($_);        #it's always a hash, 
+so expand it out (sub returns a string)
            }
            $field_data .= $data;
            next;                        #and hit the next value
        }
        if ($d_fields->{$_})                    #For dates
        {
            $field_data .= format_date(
                $tick_inf{ $hfield_names->{$_} }
                );                    #format the date (sub routine)
            next;                        #and hit the next value
        }
        if ($l_fields->{$_})                    #if the field is a sel
+ection value
        {
            $field_data .= $l_fields->{$_}->{         #change number i
+nto the name:
                    $tick_inf{            #fill the data from the $lis
+t field hash    
                    $hfield_names->{$_}         #
                        }            #
                    };
            next;
        }
        $field_data .= $tick_inf{ $hfield_names->{$_} };    #
                                    #value is in the ticket
        }#close foreach @$fields
       #more stuff which gets rid of $field_data
    }
[download]

UPDATE: I added a } to properly denote the end of the foreach (@$afields) loop. It is redefined at the beginning of each ticket (foreach sort keys %tickets) {}. Sorry about the bad formatting, this is what happens when you cut and paste code. The problem is not in the $field_data (IMHO)

Note that before running this foreach loop the memory usage is relatively low (~40 meg). As this portion of the script runs the memory continues to increase (not an increase, then drop, then increase again). After this foreach loop we dump $field_data to a file and "undef" it.

So, is there a bug that I am not seeing in there? Is there some reason that this code is chewing up so much space?

You are storing all the information in some datastructure. It just doesn't happen to be a HoLoHoL that you mentioned earlier. It's that string $field_data. Processing each ticket individually means that when you're done with the loop, there is nothing in memory but what you had before you entered the loop. :-)

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

Why don't you try writing to the file as you go and clearing $field_data, rather than storing it all up until the end? That could be the problem.

Do you get the same/similar memory usage with just a call to ars_GetEntry in the loop and no other processing? If not, then you're losing memory somewhere else in the loop.

I don't know but I am guessing that a hash has about the same efficiency.

The problem is that a deep memory structure multiplies these inefficiencies together. So you have
9000 records * 5k /.66/.66/.66/.66 = 237.2meg

Technical solutin 1
One solution is to use a flatter data structure. If you use a single-level hash with a key of Rem|Schema|Ticket it will use much less memory. Of course it will also require more code and more cpu.

Technical solution 2
Presize the arrays so that they don't allocate so much memory. This is easy for simple arrays but I have not seen it done for deeper data structures.

Political solution 1
It sounds to me like your server admins need to find more important work than making you save 200 meg of memory. Perhaps that issue is best saved for when you get asked for comments on their employee evaluation :-). More likely is that you have created a really fast application that is embarrassing the server people, and they want you to slow it down so their solutions don't look so bad in comparison. So they complain about a trivial amount of memory. If you offer to teach the server admins perl, they may stop complaining. It worked for me once!

It should work perfectly the first time! - toma

If it's difficult to get the data one record at a time, you should consider which is more expensive: a little more RAM, or a couple days of your programming time. Sometimes resource efficiency isn't the most cost-effective solution.

foreach (sort keys %tickets)
{
  my $ticket = ((split " ",$_)[0]);
  my %tick_inf;                          
  (%tick_inf = ars_GetEntry($rem,$schema,$ticket))
    || warn "Could not retrieve $ticket: $ars_errstr";
  ++$count;
  print "$count\n";
  next;
  #all the other stuff from yesterday
}
[download]

print "Script ended on " . localtime(time);
print "\n";
[download]