expresspotato has asked for the wisdom of the Perl Monks concerning the following question:

Hey Guys, Having a slight problem with threads and perl and leaking memory. Declaring a thread with 
foreach (@go_do_something){
  my $thr = threads->new(\&some_sub,$_);
  $thr -> detach();
}

[download]
Inside a loop eventually crashes the program without any error message. I can see why as defining the thread with my means the next iteration could simply get wiped out, which is probably what happens. 
foreach (@go_do_something){
  $used_threads++;
  @thr_ll[$used_threads] = threads -> new(\&some_sub,$_);
  @thr_ll[$used_threads] -> detach();
}

[download]
Does not crash the program and seems to be stable after hours of use however it is quickly and consistently leaking memory (more or less 10 MB per iteration). The subroutine some_sub can run for minutes or hours depending on how long the thread takes to run. Any help on how to either: cleanup after the thread is finished (threads->exit() within the tread has no effect on leaking memory), not have my $thr overwrite it self, or your own custom solution would be greatly appreciated. _Kevin

Replies are listed 'Best First'.
Re: Perl detached threads, loops & my?
by BrowserUk (Patriarch) on Jun 08, 2010 at 23:15 UTC

    Which OS and versionof Perl are you using? This doesn't seem to leak (with 5.10.1 & Vista): 

    perl -Mthreads -E"sub x{sleep rand 100}; for(1..100){ threads->new( \&
+x, $_ )->detach }; sleep 1000"

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Hello, I am using linux Fedora 12. The following script will consume all available memory in a matter of seconds. 
      
use threads;

while ($c < 100000){
  @thr_ll[$c]=threads->new(\&sys_call,$c);
  @thr_ll[$c]->detach();
  $c++;
}

sub sys_call(){
  print "Called: $_[0]\n";
}

      [download]
      I know the above statement is the WRONG way of using threads but its the only way I know of ensuring a fresh thread each time. It will use all system memory in a matter of seconds. With the following code the program will eventually crash after being run for extended periods (HOURS) of time with other running threads. No error message is observed. 
      use threads;

while ($c < 100000){
  my $thr=threads->new(\&sys_call,$c);
  $thr->detach();
  $c++;
}

sub sys_call(){
  print "Called: $_[0]\n";
}

      [download]
[reply]
[d/l]
[select]

        I'm sorry, but you have an unconstrained flat out loop creating hundreds of thousands of threads ... and you wonder why it runs out of memory?

        Why are you storing the thread handles when you're detaching them?

        Did you not see me ask which version of Perl you are using?

        ... but its the only way I know of ensuring a fresh thread each time.

        What on earth do you mean by a "fresh thread"? And why are you so certain that you need them?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Thank you for the example. However, I still have doubts. Please carefully consider this reply, my other reply, and BrowserUk's comments.

        Red flag for me: Your second script, you claim takes "HOURS" to run. On the below mentioned old system, that script completes for me in minutes. Process memory usage never went above 50MB.

        $ time perl script.pl
Called: 0
Called: 1
:
Called: 99998
Called: 99999

real    12m23.351s
user    10m50.800s
sys     0m23.580s

        [download]

        You're trying to create 100000 threads. You invite a race condition where your script may hit a wall very quickly if your threads do anything non-trivial. You would expect to see:

        Called: 367
Called: 368
Thread creation failed: pthread_create returned 11 at script line 4.
Can't call method "detach" on an undefined value at script line 5.

        [download]

        Your first example will "leak" memory in the sense that the @thr_ll array will grow to 100000, even after threads exit. On most systems this wouldn't be a big deal if your real application caps at 100000 as well, but I still wouldn't recommend it.

        I ran both of your scripts on my oldest machine, and neither one had a deleterious effect on Perl 5.10.1 on Ubuntu, running on an 8 year old P4-1.6GHz machine with 512MB. Both ran to completion.

[reply]
[d/l]
[select]
        I'm using a powerful computer, with 2 6-core cpus, and 8G memory: 
        $ uname -a
Linux localhost.localdomain 2.6.33.5-112.fc13.x86_64 #1 SMP Thu May 27
+ 02:28:31 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

        [download]
        with a somehow new version perl and threads: 
        $ perl -v

This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

Copyright 1987-2009, Larry Wall

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using "man perl" or "perldoc perl".  If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.

$ perl -Mthreads -e 'print $threads::VERSION,"\n"'
1.77

        [download]
        Actually, if I do something like your second example, it would core dump immediately: 
        $ perl -e 'use threads; for(1..100000) { my $t = threads->new(\&body,$
+_); $t->detach; } sub body {print "Called: @_\n"}'

        [download]
        Called: 1
Called: 2
Called: 3
Called: 4
Called: 5
Called: 6
Called: 7
Called: 8
Called: 9
Called: 10
Called: 11
Called: 12
Called: 13
Called: 14
Called: 15
Called: 16
Called: 17
Called: 18
Called: 19
Called: 20
Called: 21
Called: 22
Called: 23
Called: 24
Called: 25
Called: 26
Called: 27
Called: 28
Called: 29
Called: 30
Called: 31
Called: 32
Called: 33
Called: 34
Called: 35
Called: 36
Called: 37
Called: 38
Called: 39
Called: 40
Called: 41
Called: 42
Called: 43
Called: 44
Called: 45
Called: 46
Called: 47
Called: 48
Called: 49
Called: 50
Called: 51
Called: 52
Called: 53
Called: 54
Called: 55
Called: 56
Called: 57
Called: 58
Called: 59
Called: 60
Called: 61
Called: 62
Called: 63
Called: 64
Called: 65
Called: 66
Called: 67
Called: 68
Called: 69
Called: 70
Called: 71
Called: 72
Called: 73
Called: 74
Called: 75
Called: 76
Called: 77
Called: 78
Called: 79
Called: 80
Called: 81
Called: 82
Called: 83
Called: 84
Called: 85
Called: 86
Called: 87
Called: 88
Called: 89
Called: 90
Called: 91
Called: 92
*** glibc detected *** perl: invalid fastbin entry (free): 0x000000000
+0789010 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3b8fc75746]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89de8f]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x98)[0x3b8f89e398]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x42)[0x3b8f89e342]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f8ae131]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clean_all+0x1b)[0x3b8f8ae1db]
/usr/lib64/perl5/CORE/libperl.so(perl_destruct+0xb79)[0x3b8f84e269]
/usr/lib64/perl5/auto/threads/threads.so(+0x5496)[0x7f7b9d360496]
/usr/lib64/perl5/auto/threads/threads.so(+0x5682)[0x7f7b9d360682]
/usr/lib64/perl5/auto/threads/threads.so(+0x6849)[0x7f7b9d361849]
/lib64/libpthread.so.0[0x3b90407761]
/lib64/libc.so.6(clone+0x6d)[0x3b8fce150d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:21 3678850                         
+   /usr/bin/perl
00601000-00603000 rw-p 00001000 08:21 3678850                         
+   /usr/bin/perl
00624000-00e46000 rw-p 00000000 00:00 0                               
+   [heap]
3b8f400000-3b8f41e000 r-xp 00000000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f61e000-3b8f61f000 r--p 0001e000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f61f000-3b8f620000 rw-p 0001f000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f620000-3b8f621000 rw-p 00000000 00:00 0 
3b8f800000-3b8f962000 r-xp 00000000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8f962000-3b8fb62000 ---p 00162000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8fb62000-3b8fb6b000 rw-p 00162000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8fc00000-3b8fd75000 r-xp 00000000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8fd75000-3b8ff75000 ---p 00175000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff75000-3b8ff79000 r--p 00175000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff79000-3b8ff7a000 rw-p 00179000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff7a000-3b8ff7f000 rw-p 00000000 00:00 0 
3b90000000-3b90002000 r-xp 00000000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90002000-3b90202000 ---p 00002000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90202000-3b90203000 r--p 00002000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90203000-3b90204000 rw-p 00003000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90400000-3b90417000 r-xp 00000000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90417000-3b90616000 ---p 00017000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90616000-3b90617000 r--p 00016000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90617000-3b90618000 rw-p 00017000 08:21 3017779                   Ca
+lled: 93
Aborted (core dumped)

        [download]
[reply]
[d/l]
[select]
Re: Perl detached threads, loops & my?
by rjt (Curate) on Jun 08, 2010 at 23:59 UTC

    We need more information. What does @go_do_something contain? How many threads are you trying to create? What are they doing? In other words, can you give us a runnable but minimal example that exhibits the behavior on your system?

    Memory leak? Maybe, but, again, a real example would be necessary, here, plus some information on how you're measuring the memory and determining there is a leak.

    You said it "crashes the program without any error message". Is it possible that your script just decided it was done creating threads and exited on its own? That would produce behavior consistent with what you are seeing. Since you've detached every thread, once your @go_do_something loop exits, how do you know (in your code) that all of your threads have finished and it is safe to exit your program? What was its exit status when execution was complete? I might be completely missing the issue, here, but with the limited information available, this really seems most likely to me.

    Maybe you have a valid reason for detaching everything, but if you want to be sure all threads have finished, you might consider something along the lines of:

    use warnings;
use strict;
use threads;
use feature 'say';

my @go_do_something = (1 .. 10);

sub some_sub {
    say "Thread #$_ started";
    sleep(rand(10));
    say "Thread #$_ complete";
}

my @threads;

push @threads, threads->new(\&some_sub, $_) for @go_do_something;
say "All threads started.";
$_->join for @threads;
say "All threads joined. Exiting.";

    [download]
[reply]
[d/l]
      @BrowserUK - I am using the latest version of perl (why would anyone use anything less?)
      The only reason I "store them" is to ensure it is a fresh thread everytime. Using my $thr -> will eventually crash.
      The script is designed to do one thing. Process the list of tasks and just keep on going. Tasks (detached threads) may take a second and they may take an hour.
      I do not claim it takes "Hours to run", I claim it may be "run for hours".
      @llancet
      Ah ha! See... Always calling my $thr on occasion seems to overwrite / invalidate the older thread. When checking memory usage using my $thr, it never seems to increase which is very odd to me.
      @rjt
      Thanks for the code, however the script is "always running" and does not ever exit.
      Perhaps the actual code will shed some light on what is really going on. Its attached below. Keep in mind the same problem still exists. How to ensure this is a FRESH thread (as my or the same $thr will eventually crash), without wasting all that memory. 
      
use threads;
do("./sql.pl"); #NOT INCLUDED IN EXAMPLE

$thr4 = threads->new(\&processor);
$thr4->detach();

sub processor{
   my $select;
     $select = qq~ select * from pool_process where any="1" and status
+="" order by id asc limit 1; ~;
    
    while (1){
        if ($sys_ok == 1){
            sleep(1);
          my @row = &row_sql($select,-1);

            if (!(@row)){
         #pass
            }else{
          &del_sql(qq~ update pool_process set status = "RUNNING",proc
+_srv="$server_id" where id="@row[0]"; ~,-1);
          print "Processing (@row[1],@row[2],@row[3],@row[4])\n";
          $used_threads++;
          my $thr =threads->new(\&process_start,@row[1],@row[2],@row[3
+],@row[4],@row[5],@row[6],@row[7],@row[0]);
          $thr->detach();
           
          if (@row[4] eq "generate_details"){
           sleep(3);
          }
            }
        }else{
            sleep(30);
        }
    }    
}

sub process_start{
    my $return_code = system(qq~ perl ./thread_process.pl "$_[0]" "$_[
+1]" "$_[2]" "$_[3]" "$_[4]" ~);
}

      [download]
[reply]
[d/l]

        @expresspotato. This ain't twitter.

        What you are doing makes no sense.

        What you say you need: "FRESH threads" makes no sense.

        And if you cannot be bothered to answer my questions properly, then I can't be bothered to help.

        Good luck.

[reply]

        This is obviously neither the real code nor an example that we can run. Please, help us help you. Read the responses you've been given, and give us a minimal and runnable! example (with inputs and expected outputs) with strict and warnings that we can use to reproduce the undesired behavior that you see. Until then, we can't effectively help you.

[reply]

Back to Seekers of Perl Wisdom