Afternoon

Yeah, I get the same results (perl, v5.8.0 built for i386-linux-thread-multi; WWW::Mechanize 0.70). Like Roy Johnson suggested above, it's because WWW::Mechanize keeps a list of HTTP results in a page stack. Whenever it starts to get a new page, it stores the last response it received in an array.

package WWW::Mechanize::KeepSlim;

our @ISA = qw/WWW::Mechanize/;

sub _push_page_stack {
    my $self = shift;

    if ( $self->{res} ) {
    my $save_stack = $self->{page_stack};
    $self->{page_stack} = [];
    push( @$save_stack, $self->clone );

        # HERE! - stop the stack getting bigger than 10
    if ( @$save_stack > 10 ) {
        shift(@$save_stack);
    }

    $self->{page_stack} = $save_stack;
    }
    return 1;
}

package main;

my $agent = WWW::Mechanize::KeepSlim->new();

# ....
[download]

If you use this class with your example that demonstrates the problem, you should see the memory usage increase arithmetically for the first 10 requests, then stop increasing.

cheers
ViceRaid

It might be more useful to remove element base on its age. As the frequency you accessing web is not the same through out the day, and when you are busy surfing, you don't want the history get cleaned up quicker than the time you are mainly idle.

You may modify the structure of $self->{page_stack} a little bit, so that a time stamp is kept, and only those ones older than a certain age will get deleted.

However, as you are subclassing, it is probably a better idea to keep $self->{page_stack} as it is, and add a new array $self->{page_time_stamp}. The elements of those two arrays match 1-on-1.

The performance for deleting would be good, as the ones need to be deleted always stay together at the beginning of the array.

For future reference: You can set the max. stack depth with $mech->stack_depth(100) now:

=head2 $mech->stack_depth( $max_depth )

Get or set the page stack depth. Use this if you're doing a
lot of page scraping and running out of memory.

A value of 0 means "no history at all."  By default, the max
stack depth is humongously large, effectively keeping all
history.
[download]

Hi, ViceRaid,

Thank you very much for your explaination and code. It worked. I really appreciate.

CW

agent size = 110364
agent size = 110537
agent size = 110537
agent size = 110537
agent size = 110537
agent size = 110537
agent size = 110537
agent size = 110537
[download]

Right guess - the history feature was causing the problem, but from the Changelog for WWW::Mechanize it looks like that the page-stack feature might have been broken in 0.48 (see entry for 0.59).

cheers
ViceRaid

i.e.:

#!/usr/bin/perl

use WWW::Mechanize;
use Devel::Size qw(size total_size);


while (1) {
  my $agent = WWW::Mechanize->new();
  
  $agent->get(qq(http://www.yahoo.com));

  print "agent size = ".total_size($agent)."\n";
}
[download]

I don't believe it is leaking memory? Here's a quote from the perldoc for WWW::Mechanize.

Mech also stores a history of the URLs you've visited, which can be queried and revisited.

agent size = 123086
agent size = 245249
agent size = 367383
agent size = 489517
agent size = 611651

This shows me that the size is going up by an almost constant size each loop. I then modified your program to use Data::Dumper and spit out the results after five GETs. Here's the code.

#!/usr/bin/perl -w
                                                                      
+                                           
use strict;
use WWW::Mechanize;
use Devel::Size qw(size total_size);
use Data::Dumper;
                                                                      
+                                           
my $agent = WWW::Mechanize->new();
                                                                      
+                                           
for(my $i = 0; $i < 5; $i++) {
  $agent->get(qq(http://www.yahoo.com));
  print "agent size = ".total_size($agent)."\n";
}
                                                                      
+                                           
print Dumper($agent);
[download]

Looking at the output, there is a data structure that is part of the module called the page_stack. I'm guessing, that the implementation of the back() method uses the page_stack. That way, rather than re-requesting the pages, the pages are just reloaded from memory. I don't think this is a leak. This is just appears to be the functionality of the module.