daviddhall has asked for the wisdom of the Perl Monks concerning the following question:

The Great Monks -
I'm parsing huge log files of urls looking for specific paramter values. In an attempt to not "reinvent the wheel", I decided to use CGI.pm. But it seemed like memory was leaking like crazy.

Here's a quick example of what I'm talking about: 
use CGI;

while (<STDIN>)
{
   if (1)
   {
      my $cgi = new CGI($_);
      my $blah = $cgi->param('q')
   }
}

[download]

You can see that memory is being used about a Mb a sec. if you feed it some random text. Shouldn't the memory be freed (or atleast continued to be used by the process) rather than gargling more? (since it's local to the "if") I even tried "undef".

Also, does anyone have a better solution than using the weighty cgi.pm since all I'm really interest in is just getting the parameter values? I guess I could write some sort of regular expression but there's a lot of room for error.

Thanks in advance- David

Replies are listed 'Best First'.
Re: Memory leaking? - CGI.pm
by vladb (Vicar) on May 07, 2002 at 18:59 UTC
    Hmmm, peculiar bug.

    I always thought that CGI isn't really as clunky and resource thursty since most of it's internals are loaded on-demand at run-time. There's not even a lot of code that perl has to parse initially (most of subs are evaled 'into' the package).

    However, for the lightweight options, I suggest you look at CGI::Minimal. Give it a thorough try and see if it won't have a similar 'memory leaking' problem.

    Cheers,

    UPDATE: corrected module link *grin* ;).

    "There is no system but GNU, and Linux is one of its kernels." -- Confession of Faith
[reply]
Re: Memory leaking? - CGI.pm
by jsprat (Curate) on May 07, 2002 at 19:19 UTC
    Internally, CGI.pm uses the URI module to parse a uri. I'd use the query_form method to parse each line of the log file. Something like this: 

    use URI;

my $u = URI->new("http://perlmonks.org/index.pl?node_id=164782&abc=123
+");
my @param = $u->query_form();
foreach (@param) {
    print "$_\n";
}

    [download]

    output:
    C:\S>perl uritest.pl
node_id
164782
abc
123

    [download]

[reply]
[d/l]
[select]
Re: Memory leaking? - CGI.pm
by shotgunefx (Parson) on May 07, 2002 at 19:28 UTC
    What happens if you force it to use the default object? 
    if (1){
   $CGI::Q = new CGI($_);
   my $blah = param('q');
}

    [download]
    I'd try it myself but can't at the moment.

    -Lee

    "To be civilized is to deny one's nature."
[reply]
[d/l]
      It seems like all of these continued to eat more memory. Curious though that the URI did as well. Maybe I'm just confused on local scope and memory freeing?
[reply]
Re: Memory leaking? - CGI.pm
by jlongino (Parson) on May 08, 2002 at 01:02 UTC
    I'm not sure what version (perl, CGI) you're running, but I don't seem to be able to duplicate the problem in either of my two environments: 
       WinXP:
perl, v5.6.1 built for MSWin32-x86-multi-thread
Binary build 631 provided by ActiveState Tool Corp. Built 17:16:22 Jan
+  2 2002
$CGI::revision = '$Id: CGI.pm,v 1.49 2001/02/04 23:08:39 lstein Exp $'
+;
$CGI::VERSION='2.752';

   Solaris 8:
perl, v5.6.0 built for sun4-solaris
$CGI::revision = '$Id: CGI.pm,v 1.19 1999/08/31 17:04:37 lstein Exp $'
+;
$CGI::VERSION='2.56';

    [download]
    Neither of the boxen had leaks running the following: 
    use strict;
use CGI;
$|++;

my $ct = 0;
while (1)
{
   $ct++;
   if (1)
   {
      my $cgi = new CGI("q=$ct");
      my $blah = $cgi->param('q');
      print " blah: $blah\n";
   }
}

    [download]
    On the PC I let them run for more than 256 seconds (256 MB RAM) and I monitored the Solaris version using top with no visible memory leaks. You might try running my code and see if it leaks as well, but I don't think CGI.pm is the cause.

    --Jim

[reply]
[d/l]
[select]

      Removing the `print' command, and running on Solaris, definitely shows a memory leak (at around 0.4MB/sec) if I run with Perl 5.005. 

      % uname -a
SunOS mir 5.8 Generic_108528-07 sun4u sparc SUNW,Ultra-2
% perl -MCGI -le 'print $CGI::VERSION'
2.46

      [download]
      Running with Perl 5.6.1 on the same platform, I observe no memory leak. This is strange.

[reply]
[d/l]
      I ran your code, and it definetly shows memory disappearing...

      BTW, here's the version info for my perl:
      "This is perl, version 5.005_02 built for alpha-dec_osf"

      I guess maybe from what the other guy posted things might be better with a newer version of perl.
[reply]
        BTW, if upgrading your version of Perl is a problem (some work environments are particular about that sorta thing), you might consider just installing a newer version of CGI.pm and see if that helps first. I don't know what the ramifciations of the CGI-only upgrade would be--maybe you could start a separate thread on that. I'd be interested in reading the replies, just out of curiosity.

        --Jim

        Update: make sure the version of CGI.pm you install is compatible with 5.005_02 built for alpha-dec_osf.

[reply]

Back to Seekers of Perl Wisdom