perl 5.12.3 + threads + memory leak

kamenpetrov has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks !

I need your suggestion on something I can't figure out.

So, I am running:
/usr/local/bin/perl -v
This is perl 5, version 12, subversion 3 (v5.12.3) built for i686-linux

and I have the following simple script:

#!/usr/local/bin/perl
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(2);

$page = 'http://www.sics.se/~joe/bluetail/vol1/v1_oo.html';
use threads;

$total = 0;
while (1==1) {
        if ($total < 50) {
                $total++;
                my $threadname = 'thr'.$total;
                $$threadname = threads->create(\&sub1, $page);
        }
        foreach my $thread_join (threads->list(threads::joinable)) {
                my $res = $thread_join->join();
                print "RES:[$res]\n";
        }
        sleep(1);
}
sub sub1() {
        my ($url) = @_;
        print "\t FETCHED $url\n";
        my $response = $ua->get($url);
        if ($response->is_success) { $content = $response->decoded_con
+tent; }
        else { $content = ''; }

        return length($content);
}
[download]

So.. that thing is leaking memory :(
The script is really simple, but not that simple for me to figure out the cause for the leak.

What I suspect is.. eventually IO::Socket does it...

Any suggestions are welcome.

Thank you in advance.

Comment on perl 5.12.3 + threads + memory leak Download Code

Replies are listed 'Best First'.
Re: perl 5.12.3 + threads + memory leak by zwon (Abbot) on Feb 14, 2011 at 15:39 UTC
http://rt.perl.org/rt3/Ticket/Display.html?id=69598 It is fixed in 5.13, but not in 5.12. As alternative, I can recommend AnyEvent::HTTP, which on my taste is much better than using threads.	[reply]
Re^2: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 15, 2011 at 10:45 UTC
Thanks for the info. I have tested the following scenarios: - 5.10.0/5.12.3 + LWP --> both LWP and Perl leaks memory - 5.10.0/5.12.3 + curl --> Perl leaks memory - 5.13.9 + LWP --> LWP leaks memory - 5.13.9 + curl --> no leak Thank you once again for your time and help!	[reply]
Re^3: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 17, 2011 at 00:43 UTC
Ok, now to the real problem: - perl 5.13.9 - proc::daemon - threads - DBD::Pg So, I have a script running as a daemon. The daemon listen to a Postgres database for "notify" events. Once there is new event, the daemon fetch all records from a database table, passes them back to a database procedure (equal to SELECT statement) and the result is passed as string to another sub routine in the perl script. The routine generates an URL and that URL is fetched via curl in a perl thread. All that works good, except the fact DBD::Pg leaks memory. All the code is "strict"/"warnings"-safe and the Pg module leaks really serious amount of memory. The module I pick for tracing the leaks returns many records in the DBI.pm. Looking trough DBD::Pg code, I noticed the CLONE function does not really do what it is supposed to do (compare to DBD::mysql). It only exists so to stop the DBI module from reporting warnings. Does anyone of you guys have used DBD::Pg in a multi-threaded script and how you manage the memory leaks? Thanks	[reply]
Re^4: perl 5.12.3 + threads + memory leak by BrowserUk (Patriarch) on Feb 17, 2011 at 02:00 UTC
Re^5: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 17, 2011 at 12:24 UTC
Some notes below your chosen depth have not been shown here
Re: perl 5.12.3 + threads + memory leak by BrowserUk (Patriarch) on Feb 14, 2011 at 15:16 UTC
I'm not surprised at all looking at your code. Re-post when you've made it strict-safe and I'll take a look. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^2: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 14, 2011 at 15:25 UTC
That should be strict-safe and still leaking memory: #!/usr/local/bin/perl use strict; my ($page,$total) = ('http://www.sics.se/~joe/bluetail/vol1/v1_oo.html +',0); use threads; for (my $i=0;$i<10;$i++) { threads->create({'stack_size' => 32*4096},\ +&sub1, $page, $i); } while ($total < 10) { foreach my $thread_join (threads->list(threads::joinable)) { $total++; my $res = $thread_join->join(); print "Thread done\n"; } } print "Press ENTER to finish"; my $ok = <STDIN>; sub sub1() { my ($url,$i, $content) = @_; require LWP::UserAgent; my $ua = LWP::UserAgent->new; my $response = $ua->get($url); print "\t[$i]FETCHED $url\n"; if ($response->is_success) { $content = $response->decoded_con +tent; } return; } [download] Thanks in advance.	[reply] [d/l]
Re^3: perl 5.12.3 + threads + memory leak by BrowserUk (Patriarch) on Feb 14, 2011 at 16:16 UTC
If this leaks on your system (it doesn't on mine) then zwon has called it below: `perl -Mthreads -e'async{ async{ 1;}->join }->join while 1'` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re: perl 5.12.3 + threads + memory leak by Anonymous Monk on Feb 14, 2011 at 14:11 UTC
Try `sub sub1() { require LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->timeout(2); ...` [download]	[reply] [d/l]
Re^2: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 14, 2011 at 14:19 UTC
I just tried that but the leak remains.	[reply]
Re^3: perl 5.12.3 + threads + memory leak by Anonymous Monk on Feb 14, 2011 at 14:27 UTC
Get rid of `$$threadname`	[reply] [d/l]
Re^4: perl 5.12.3 + threads + memory leak by kamenpetrov (Novice) on Feb 14, 2011 at 14:30 UTC
Re: perl 5.12.3 + threads + memory leak by OlegG (Monk) on Feb 16, 2011 at 05:19 UTC
use Coro and Coro::LWP. It is really simple and effectively.	[reply]