submersible_toaster has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks
I discovered what I assumed to be a memory leak in an application I am writing using CGI::Application as a base. Initially I blamed my OO style - believing that some references were circular but this turned out to be untrue (I think).

The problem arises when using CGI::Application to return data to the browser. I used the following as a method to deliver images. 

sub getimage {
 my $self = shift;
 my $stuff = shift;
 # Set the correct header
 $self->header_props( -type=>$stuff->{mime} );
 open ( IMAGE , $stuff->{image} ) or die "Screaming $!";
 return join '', <IMAGE>;
}

[download]

Which functioned beautifully (once PodMaster slapped some sense into me regarding returning scalars to CGI:Application) however my httpd processes grew at an alarming rate. For every 1Mb image served , httpd grows by a larger amount.


Clean startup 
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
18388 root      15   0  2848 2848  2736 S     0.0  0.5   0:00 httpd
18389 vvr       18   0  2836 2836  2716 S     0.0  0.5   0:00 httpd
18390 vvr       18   0  2836 2836  2716 S     0.0  0.5   0:00 httpd

After serving one 6.2Mb image 
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
18388 root      15   0  2848 2848  2736 S     0.0  0.5   0:00 httpd
18389 vvr       15   0 31940  31M  2372 S     0.0  6.2   0:00 httpd
18390 vvr       18   0  2836 2836  2716 S     0.0  0.5   0:00 httpd

The resident memory size has grown from under 3Mb to over 31Mb. Almost 5 times the size of the data actually served. I am most curious to hear from anyone who has experienced similar behaviour (with CGI::Application especially).

This is occuring on a RH7.3 system, using apache1.3.26 and mod_perl 1.27. Below is the smallest case that exhibits this behaviour.

#!/usr/bin/perl -w
# foo.pm , to be executed by Apache
package foo;
use strict;
use base 'CGI::Application';
use CGI::Carp;

sub handler {
 warn "## handler";
 my $request = shift;
 my $app = foo->new();
 $app->run();
 exit;
}

sub setup {
 warn "## setup";
 my $self = shift;
 $self->run_modes(
  start=>'start'
 );
}


sub start {
 warn "## start";
 my $self = shift;
 open ( FH , '/path/to/some/image' ) or confess $!;
 $self->header_props( -type=>'your/mimetype' );
 return join '' , <FH>;
}


1;

[download]

httpd.conf
PerlModule Apache
PerlModule foo

<Location /testleak/>
 SetHandler perl-script
 PerlHandler foo
</Location>

[download]

If you've solved this before, or think you can see where I have done something bass-ackwards please let me know.


Many Thanks -
toaster.

I can't believe it's not psellchecked

Replies are listed 'Best First'.
Re: CGI::Application + mod_perl =|!= memory leak
by diotalevi (Canon) on May 21, 2003 at 07:08 UTC

    You goofed and forgot to binmode IMAGE or die "Couldn't binmode image". Always do that when working with binary data. You also never close FH. If you meant to just return the contents of the file then you're better off writing that as a plain read - you're forcing perl to jump through hoops.

    # Ok, could be better
local $/;
open ( FH , '/path/to/some/image' ) or confess $!;
binmode FH or confess $!;
my $image = readline *FH;
close FH or confess $!;
return $image;

# Better
open ( FH , '/path/to/some/image' ) or confess $!;
binmode FH or confess $!;
my $image;
read FH, $image, -s FH;
close FH or confess $!;
return $image;

# Best
open ( FH , '/path/to/some/image' ) or confess $!;
my $image;
read FH, $image, -s FH;
close FH or confess $!;
return \ $image;

    [download]
[reply]
[d/l]
[select]

      binmode having no use (purportedly) under linux I did not use it. I should have noted that the shortest case foo.pm is really the shortest case, that includes removing the close on the filehandle (which is going out of scope anyway), and also returning a reference to a scalar rather than the scalar itself.

      Sadly, none of these sensible solutions makes the leak go awayl, but I think that returning the reference kept two copies of $image in memory - one copy dissapears once the request in completed.

      diotalevi++
      thankyou

      I can't believe it's not psellchecked
[reply]
[d/l]
        No, "globbish" filehandles are package variables and never go out of scope. The "best" variant would look like so: 
        sub start {
    warn "## start";
    my $self = shift;
    open my $fh, '<', '/path/to/some/image' or confess $!;
    $self->header_props(-type=>'your/mimetype');
    local $/;
    return <$fh>;
}

        [download]
        Though that does of course do nothing to solve your leak.

        Makeshifts last the longest.

[reply]
[d/l]

        If I recall correctly, binmode is something that unix denizens are going to have to use religiously in the future for the perl5 line. So far it isn't a big deal for Linux folk but I seem to recall that around 5.10.x it'll be something you'll need to have done.

        Anyway, its only proper to flag your filehandle so it can handle the right data appropriately.

[reply]
Re: CGI::Application + mod_perl =|!= memory leak
by perrin (Chancellor) on May 21, 2003 at 15:20 UTC
    First, you need to understand the implications of a persistent Perl interpreter. When Perl loads something into a lexical variable and the lexical goes out of scope, it doesn't release that memory. This is an optimization, assuming you will probably use that variable again. What this means is that if you load a big file into memory (and remember that Perl takes a lot more RAM to store something than it takes on disk), you will permanently add the size of that file to your process.

    However, you should not see any growth if you just keep loading the same file over and over again. If that's what you see, then you may have a bug somewhere.

    The second thing to understand is that this is probably a bad idea. Why serve a static image from your dynamic script? Why not simply redirect or serve it straight from apache?

    If you really must serve the image yourself, look at Apache::File. There's an example of using it in the Eagle book.

[reply]
Re: CGI::Application + mod_perl =|!= memory leak
by PodMaster (Abbot) on May 21, 2003 at 08:25 UTC
    I'm running ActivePerl 5.6.1 Build 633 on Win2000 SP3, with 
    Server version: Apache/1.3.27 (Win32)
Server built:   Oct  3 2002 18:55:42
Server's Module Magic Number: 19990320:13
Server compiled with....
 -D HAVE_MMAP
 -D USE_MMAP_SCOREBOARD
 -D NO_WRITEV
 -D NO_OTHER_CHILD
 -D NO_RELIABLE_PIPED_LOGS
 -D HARD_SERVER_LIMIT=1024
 -D MULTITHREAD
    with MOD_PERL = mod_perl/1.27_01-dev

    and I tried your code as you had it set up, and I was getting leaks.
    I also tried it under Apache::Registry, and I was getting leaks.
    I also tried it as straight cgi, and still, I was getting leaks.
    Then I thought, hmm, so I tried even the simplest of cgi programs under cgi/mod_perl and I was getting leaks. I'll try Apache2 later, maybe my luck will change and i'll be more helpfull.

    I suggest you try running it under Apache::Registry and as straight CGI, and then inquire on the module's mailing list cgiapp-subscribe@lists.erlbaum.net and maybe the mod_perl mailing list.

    Good luck.

    update: btw, perl -MCGI::Application -e"die $CGI::Application::VERSION" yields [Thu May 22 00:20:15 2003] -e: 3.0 at -e line 1.

    MJD says you can't just make shit up and expect the computer to know what you mean, retardo!
    I run a Win32 PPM repository for perl 5.6x+5.8x. I take requests.
    ** The Third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]
[select]
Re: CGI::Application + mod_perl =|!= memory leak
by cees (Curate) on May 21, 2003 at 14:06 UTC

    I tested your code exactly as you posted it on mod_perl 1.99_09 and I did not notice any leaks.

    Apache/2.0.45 (Unix) mod_perl/1.99_09 Perl/v5.8.0 DAV/2
CGI::Application v2.6

    Thought you might be interested to know...

[reply]

Back to Seekers of Perl Wisdom