in reply to Re: Memory usage & hashes of lists of hashes of lists
in thread Memory usage & hashes of lists of hashes of lists

I am processing them one by one, it appears that perl keeps chewing space anyways. Here's the relevant section of code:
foreach (sort keys %tickets) { my $ticket = ((split " ",$_)[0]); my %tick_inf; #hash to hold ticket info (%tick_inf = ars_GetEntry($rem,$schema,$ticket)) || warn "Could not retrieve $ticket: $ars_errstr"; #populat +e the hash with the ticket info my $field_data=$ticket; foreach (@$afield_names){ #these are the fields that we have be +en asked to grab. $field_data .= " " . $sep; #join the previous d +ata with the seperator unless (exists $hfield_names->{$_}) { print "Bad field name -- $_\n"; next; } unless (exists $tick_inf{$hfield_names->{$_}}) { print $hfield_names->{$_} . "\n"; print "Bad ticket info identifier -- $_ does not relate to +: " . $hfield_names->{$_} . "\n"; next; } unless ( defined($tick_inf{$hfield_names->{$_}})) { $field_data .= " "; next; } if (ref($tick_inf{ $hfield_names->{$_} }) eq "ARRAY") #For +array references { my $data; foreach (@{ $tick_inf{ $hfield_names->{$_} } }) #foreach o +f the values of the array ref { $data .= expand_hash($_); #it's always a hash, +so expand it out (sub returns a string) } $field_data .= $data; next; #and hit the next value } if ($d_fields->{$_}) #For dates { $field_data .= format_date( $tick_inf{ $hfield_names->{$_} } ); #format the date (sub routine) next; #and hit the next value } if ($l_fields->{$_}) #if the field is a sel +ection value { $field_data .= $l_fields->{$_}->{ #change number i +nto the name: $tick_inf{ #fill the data from the $lis +t field hash $hfield_names->{$_} # } # }; next; } $field_data .= $tick_inf{ $hfield_names->{$_} }; # #value is in the ticket }#close foreach @$fields #more stuff which gets rid of $field_data }
UPDATE: I added a } to properly denote the end of the foreach (@$afields) loop. It is redefined at the beginning of each ticket (foreach sort keys %tickets) {}. Sorry about the bad formatting, this is what happens when you cut and paste code. The problem is not in the $field_data (IMHO)

Note that before running this foreach loop the memory usage is relatively low (~40 meg). As this portion of the script runs the memory continues to increase (not an increase, then drop, then increase again). After this foreach loop we dump $field_data to a file and "undef" it.

So, is there a bug that I am not seeing in there? Is there some reason that this code is chewing up so much space?

Replies are listed 'Best First'.
Re: Re: Re: Memory usage & hashes of lists of hashes of lists
by dragonchild (Archbishop) on Sep 28, 2001 at 02:05 UTC
    You are storing all the information in some datastructure. It just doesn't happen to be a HoLoHoL that you mentioned earlier. It's that string $field_data. Processing each ticket individually means that when you're done with the loop, there is nothing in memory but what you had before you entered the loop. :-)

    ------
    We are the carpenters and bricklayers of the Information Age.

    Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

      But, the $field_data variable is undefed at the end of the loop @$fields loop (see my update). The @$afields array is only retrieving the 90 fields per ticket. Again, at max that field is about 3k at the end, so it's not an ever increasing variable - also note how it is redefined at the start of every new ticket (below first foreach), so I really don't think that $field_data is causing the memory issues (yes it's not the most efficient way of doing things, but it is NOT sucking up 150 meg of RAM on the server).
Re: Re: Re: Memory usage & hashes of lists of hashes of lists
by perrin (Chancellor) on Sep 28, 2001 at 01:44 UTC
    Why don't you try writing to the file as you go and clearing $field_data, rather than storing it all up until the end? That could be the problem.
Re: Re: Re: Memory usage & hashes of lists of hashes of lists
by runrig (Abbot) on Sep 28, 2001 at 01:58 UTC
    Do you get the same/similar memory usage with just a call to ars_GetEntry in the loop and no other processing? If not, then you're losing memory somewhere else in the loop.