in reply to Perl detached threads, loops & my?

Which OS and versionof Perl are you using? This doesn't seem to leak (with 5.10.1 & Vista): 

perl -Mthreads -E"sub x{sleep rand 100}; for(1..100){ threads->new( \&
+x, $_ )->detach }; sleep 1000"

[download]
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^2: Perl detached threads, loops & my?
by expresspotato (Beadle) on Jun 09, 2010 at 00:00 UTC
    Hello, I am using linux Fedora 12. The following script will consume all available memory in a matter of seconds. 
    
use threads;

while ($c < 100000){
  @thr_ll[$c]=threads->new(\&sys_call,$c);
  @thr_ll[$c]->detach();
  $c++;
}

sub sys_call(){
  print "Called: $_[0]\n";
}

    [download]
    I know the above statement is the WRONG way of using threads but its the only way I know of ensuring a fresh thread each time. It will use all system memory in a matter of seconds. With the following code the program will eventually crash after being run for extended periods (HOURS) of time with other running threads. No error message is observed. 
    use threads;

while ($c < 100000){
  my $thr=threads->new(\&sys_call,$c);
  $thr->detach();
  $c++;
}

sub sys_call(){
  print "Called: $_[0]\n";
}

    [download]
[reply]
[d/l]
[select]

      I'm sorry, but you have an unconstrained flat out loop creating hundreds of thousands of threads ... and you wonder why it runs out of memory?

      Why are you storing the thread handles when you're detaching them?

      Did you not see me ask which version of Perl you are using?

      ... but its the only way I know of ensuring a fresh thread each time.

      What on earth do you mean by a "fresh thread"? And why are you so certain that you need them?

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      Thank you for the example. However, I still have doubts. Please carefully consider this reply, my other reply, and BrowserUk's comments.

      Red flag for me: Your second script, you claim takes "HOURS" to run. On the below mentioned old system, that script completes for me in minutes. Process memory usage never went above 50MB.

      $ time perl script.pl
Called: 0
Called: 1
:
Called: 99998
Called: 99999

real    12m23.351s
user    10m50.800s
sys     0m23.580s

      [download]

      You're trying to create 100000 threads. You invite a race condition where your script may hit a wall very quickly if your threads do anything non-trivial. You would expect to see:

      Called: 367
Called: 368
Thread creation failed: pthread_create returned 11 at script line 4.
Can't call method "detach" on an undefined value at script line 5.

      [download]

      Your first example will "leak" memory in the sense that the @thr_ll array will grow to 100000, even after threads exit. On most systems this wouldn't be a big deal if your real application caps at 100000 as well, but I still wouldn't recommend it.

      I ran both of your scripts on my oldest machine, and neither one had a deleterious effect on Perl 5.10.1 on Ubuntu, running on an 8 year old P4-1.6GHz machine with 512MB. Both ran to completion.

[reply]
[d/l]
[select]
      I'm using a powerful computer, with 2 6-core cpus, and 8G memory: 
      $ uname -a
Linux localhost.localdomain 2.6.33.5-112.fc13.x86_64 #1 SMP Thu May 27
+ 02:28:31 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

      [download]
      with a somehow new version perl and threads: 
      $ perl -v

This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

Copyright 1987-2009, Larry Wall

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using "man perl" or "perldoc perl".  If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.

$ perl -Mthreads -e 'print $threads::VERSION,"\n"'
1.77

      [download]
      Actually, if I do something like your second example, it would core dump immediately: 
      $ perl -e 'use threads; for(1..100000) { my $t = threads->new(\&body,$
+_); $t->detach; } sub body {print "Called: @_\n"}'

      [download]
      Called: 1
Called: 2
Called: 3
Called: 4
Called: 5
Called: 6
Called: 7
Called: 8
Called: 9
Called: 10
Called: 11
Called: 12
Called: 13
Called: 14
Called: 15
Called: 16
Called: 17
Called: 18
Called: 19
Called: 20
Called: 21
Called: 22
Called: 23
Called: 24
Called: 25
Called: 26
Called: 27
Called: 28
Called: 29
Called: 30
Called: 31
Called: 32
Called: 33
Called: 34
Called: 35
Called: 36
Called: 37
Called: 38
Called: 39
Called: 40
Called: 41
Called: 42
Called: 43
Called: 44
Called: 45
Called: 46
Called: 47
Called: 48
Called: 49
Called: 50
Called: 51
Called: 52
Called: 53
Called: 54
Called: 55
Called: 56
Called: 57
Called: 58
Called: 59
Called: 60
Called: 61
Called: 62
Called: 63
Called: 64
Called: 65
Called: 66
Called: 67
Called: 68
Called: 69
Called: 70
Called: 71
Called: 72
Called: 73
Called: 74
Called: 75
Called: 76
Called: 77
Called: 78
Called: 79
Called: 80
Called: 81
Called: 82
Called: 83
Called: 84
Called: 85
Called: 86
Called: 87
Called: 88
Called: 89
Called: 90
Called: 91
Called: 92
*** glibc detected *** perl: invalid fastbin entry (free): 0x000000000
+0789010 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3b8fc75746]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89de8f]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x98)[0x3b8f89e398]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x42)[0x3b8f89e342]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861]
/usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112]
/usr/lib64/perl5/CORE/libperl.so[0x3b8f8ae131]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clean_all+0x1b)[0x3b8f8ae1db]
/usr/lib64/perl5/CORE/libperl.so(perl_destruct+0xb79)[0x3b8f84e269]
/usr/lib64/perl5/auto/threads/threads.so(+0x5496)[0x7f7b9d360496]
/usr/lib64/perl5/auto/threads/threads.so(+0x5682)[0x7f7b9d360682]
/usr/lib64/perl5/auto/threads/threads.so(+0x6849)[0x7f7b9d361849]
/lib64/libpthread.so.0[0x3b90407761]
/lib64/libc.so.6(clone+0x6d)[0x3b8fce150d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:21 3678850                         
+   /usr/bin/perl
00601000-00603000 rw-p 00001000 08:21 3678850                         
+   /usr/bin/perl
00624000-00e46000 rw-p 00000000 00:00 0                               
+   [heap]
3b8f400000-3b8f41e000 r-xp 00000000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f61e000-3b8f61f000 r--p 0001e000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f61f000-3b8f620000 rw-p 0001f000 08:21 3017770                     
+   /lib64/ld-2.12.so
3b8f620000-3b8f621000 rw-p 00000000 00:00 0 
3b8f800000-3b8f962000 r-xp 00000000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8f962000-3b8fb62000 ---p 00162000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8fb62000-3b8fb6b000 rw-p 00162000 08:21 3727883                     
+   /usr/lib64/perl5/CORE/libperl.so
3b8fc00000-3b8fd75000 r-xp 00000000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8fd75000-3b8ff75000 ---p 00175000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff75000-3b8ff79000 r--p 00175000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff79000-3b8ff7a000 rw-p 00179000 08:21 3017771                     
+   /lib64/libc-2.12.so
3b8ff7a000-3b8ff7f000 rw-p 00000000 00:00 0 
3b90000000-3b90002000 r-xp 00000000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90002000-3b90202000 ---p 00002000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90202000-3b90203000 r--p 00002000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90203000-3b90204000 rw-p 00003000 08:21 3017773                     
+   /lib64/libdl-2.12.so
3b90400000-3b90417000 r-xp 00000000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90417000-3b90616000 ---p 00017000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90616000-3b90617000 r--p 00016000 08:21 3017779                     
+   /lib64/libpthread-2.12.so
3b90617000-3b90618000 rw-p 00017000 08:21 3017779                   Ca
+lled: 93
Aborted (core dumped)

      [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom