Again out of memory

pradeep has asked for the wisdom of the Perl Monks concerning the following question:

here i am finding the service before hand and then i am updating a hash for each service and for each cookie.
I am working on the log files

       if($service ne '')
          {
             if($cookie ne '')
             {
                if($loginstatus == 1) {
                   $CookHash{$cookie}{$service}->[0]++; }       # 0 - 
+logged in page views
                else {
                   $CookHash{$cookie}{$service}->[1]++; }       # 1 - 
+not logged in page views
             }
             else
             {
                $NoCookieCnt++;
                $NoCookie{$service}->[0]++;                     # No c
+ookie page views of service
             }
             $service = '';
          }
[download]

The code worked well for a couple of log files but is faltering with more files, citing out of memory.
no other error meessages and it is working properly for fewer log files

edit (broquaint): added <code> tags + formatting

Comment on Again out of memory Select or Download Code

Replies are listed 'Best First'.
Re: Again out of memory by rdfield (Priest) on Oct 30, 2002 at 09:24 UTC
You might want to have a quick read about formatting nodes. Specifically the <code> tags. You might want to put some debugging code into your application to check that the values of `$service` and `$cookie` are being set correctly. If these values are incorrect your hash might well spiral out of control. rdfield	[reply] [d/l] [select]
Re: Re: Again out of memory by derby (Abbot) on Oct 30, 2002 at 13:01 UTC
++rdfield. Without knowledge of the size of the log files all I can see is it is quite possible you are truly running out of memory. You could try, every so often, printing out your hash in scalar context ( `print scalar( %CookHash ), "\n";`) to see how hash buckets are being allocated. You may be near the end and that last hash allocation is sending you over the edge. There are a couple of small things you could do: 1. pre-allocate the hash if you know exactly how big the hash will be. 2. run against a small set of log files and produce some type of results for those and then run another script that sums up the initial results 3. if you have enough diskspace, use a Tie::Hash or a DB_File or database (DBI) to hold the information. -derby	[reply] [d/l]