in reply to Memory usage & hashes of lists of hashes of lists

Instead of storing all the data for all the tickets before you process them, why not get the ticket numbers, then loop over them and actually process each ticket within the loop. Store the actual ticket data within a lexical. Each ticket will then use the same physical memory location, thus keeping your overhead low.

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

  • Comment on Re: Memory usage & hashes of lists of hashes of lists

Replies are listed 'Best First'.
Re: Re: Memory usage & hashes of lists of hashes of lists
by the_slycer (Chaplain) on Sep 28, 2001 at 01:39 UTC
    I am processing them one by one, it appears that perl keeps chewing space anyways. Here's the relevant section of code:
    foreach (sort keys %tickets) { my $ticket = ((split " ",$_)[0]); my %tick_inf; #hash to hold ticket info (%tick_inf = ars_GetEntry($rem,$schema,$ticket)) || warn "Could not retrieve $ticket: $ars_errstr"; #populat +e the hash with the ticket info my $field_data=$ticket; foreach (@$afield_names){ #these are the fields that we have be +en asked to grab. $field_data .= " " . $sep; #join the previous d +ata with the seperator unless (exists $hfield_names->{$_}) { print "Bad field name -- $_\n"; next; } unless (exists $tick_inf{$hfield_names->{$_}}) { print $hfield_names->{$_} . "\n"; print "Bad ticket info identifier -- $_ does not relate to +: " . $hfield_names->{$_} . "\n"; next; } unless ( defined($tick_inf{$hfield_names->{$_}})) { $field_data .= " "; next; } if (ref($tick_inf{ $hfield_names->{$_} }) eq "ARRAY") #For +array references { my $data; foreach (@{ $tick_inf{ $hfield_names->{$_} } }) #foreach o +f the values of the array ref { $data .= expand_hash($_); #it's always a hash, +so expand it out (sub returns a string) } $field_data .= $data; next; #and hit the next value } if ($d_fields->{$_}) #For dates { $field_data .= format_date( $tick_inf{ $hfield_names->{$_} } ); #format the date (sub routine) next; #and hit the next value } if ($l_fields->{$_}) #if the field is a sel +ection value { $field_data .= $l_fields->{$_}->{ #change number i +nto the name: $tick_inf{ #fill the data from the $lis +t field hash $hfield_names->{$_} # } # }; next; } $field_data .= $tick_inf{ $hfield_names->{$_} }; # #value is in the ticket }#close foreach @$fields #more stuff which gets rid of $field_data }
    UPDATE: I added a } to properly denote the end of the foreach (@$afields) loop. It is redefined at the beginning of each ticket (foreach sort keys %tickets) {}. Sorry about the bad formatting, this is what happens when you cut and paste code. The problem is not in the $field_data (IMHO)

    Note that before running this foreach loop the memory usage is relatively low (~40 meg). As this portion of the script runs the memory continues to increase (not an increase, then drop, then increase again). After this foreach loop we dump $field_data to a file and "undef" it.

    So, is there a bug that I am not seeing in there? Is there some reason that this code is chewing up so much space?
      You are storing all the information in some datastructure. It just doesn't happen to be a HoLoHoL that you mentioned earlier. It's that string $field_data. Processing each ticket individually means that when you're done with the loop, there is nothing in memory but what you had before you entered the loop. :-)

      ------
      We are the carpenters and bricklayers of the Information Age.

      Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

        But, the $field_data variable is undefed at the end of the loop @$fields loop (see my update). The @$afields array is only retrieving the 90 fields per ticket. Again, at max that field is about 3k at the end, so it's not an ever increasing variable - also note how it is redefined at the start of every new ticket (below first foreach), so I really don't think that $field_data is causing the memory issues (yes it's not the most efficient way of doing things, but it is NOT sucking up 150 meg of RAM on the server).
      Why don't you try writing to the file as you go and clearing $field_data, rather than storing it all up until the end? That could be the problem.
      Do you get the same/similar memory usage with just a call to ars_GetEntry in the loop and no other processing? If not, then you're losing memory somewhere else in the loop.