problems with fork from thread

cristi1979 has asked for the wisdom of the Perl Monks concerning the following question:

Hi, can someone explain what happens when you fork from a thread? In my case the script will get stuck. Sample code that will not finish:

#!/usr/bin/perl
use threads;

sub something_1 {
  my $thr;
  foreach (1..10){
    my $q = threads->create(\&something_2, $_);
    $thr->{$_} = $q;
  }
  while (1) {
    foreach (keys %$thr) {
      if ($thr->{$_}->is_joinable()){
    $thr->{$_}->join();
    delete $thr->{$_};
      }
    }
    print "trying:". (scalar keys %$thr)."\n";
    last if not scalar keys %$thr;
    sleep 1;
  }
}

sub something_2 {
  my $a = shift;
  my $pid = fork();
  if ($pid==0) {
    print "cocot $a\n";
    exit(0);
  } else {
    waitpid($pid, 0);
    print "done $a\n";
  }
}

# my $q = threads->create(\&something_1);
something_1;
[download]

Also I think I understand the following error, but can I get a clean exit somehow?

Perl exited with active threads:
    9 running and unjoined
    0 finished and unjoined
    0 running and detached
[download]

EDIT post a simpler example. the original one was with a fork from a thread from a thread

Comment on problems with fork from thread Select or Download Code

Replies are listed 'Best First'.
Re: problems with fork from thread by DrManhattan (Chaplain) on Mar 01, 2012 at 16:43 UTC
I don't know the answer to your question, but this article seems relevant: http://stackoverflow.com/questions/1235516/fork-in-multi-threaded-program. If you just need some simple code to do 10 things at the same time, you might like Parallel::ForkManager. Example code looks something like this: `use Parallel::ForkManager; my $pm = new Parallel::ForkManager(10); foreach (1..10) { # Forks and returns the pid for the child: my $pid = $pm->start and next; &something_1(); $pm->finish; # Terminates the child process } $pm->wait_all_children;` [download] -Matt	[reply] [d/l]
Re^2: problems with fork from thread by cristi1979 (Novice) on Mar 01, 2012 at 17:49 UTC
Forking is not an ideal solution, because I exchange data between the threads and I will have to rewrite a lot to go with forks. If I use only threads, I have no problems. My problem is that I'm using n threads doing work with HTML::TreeBuilder. Somewhere in there I have some memory leaks, because the memory used by the script is increasing until it will be killed because of OOM. So my solution (that will not involve rewriting the entire application with forks) was to fork only the HTML::TreeBuilder part and so I will get read of the accumulated memory leaks. Seems I hit an other problem and I have to look on a way to go with fork and sockets. I still hope that a solution will appear for this.	[reply]
Re^3: problems with fork from thread by BrowserUk (Patriarch) on Mar 02, 2012 at 07:06 UTC
May I ask if you are successfully using threading for your program, why to you want to fork as well? What are you going to do after you fork? Note: I'm not condemning the use of fork here, just trying to see a way to a solution for you that avoids needing to re-write your entire application. If, for example, you need to run an external command, then there are a couple of ways of approaching that that are known to work. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^4: problems with fork from thread by cristi1979 (Novice) on Mar 02, 2012 at 07:40 UTC
Re^5: problems with fork from thread by Gangabass (Vicar) on Mar 04, 2012 at 02:49 UTC
Some notes below your chosen depth have not been shown here
Re^3: problems with fork from thread by Gangabass (Vicar) on Mar 02, 2012 at 06:42 UTC
I have same problem recently (threaded app with `HTML::TreeBuilder::XPath`). I have downloaded and processed less than 6000 entries (html pages) but my program uses 2Gb! I have fixed this issue by adding `$tree->delete()` after I don't use it. Thanks. Roman	[reply] [d/l] [select]
Re: problems with fork from thread by cristi1979 (Novice) on Mar 01, 2012 at 15:18 UTC
So it seems if I can get a clean exit from the forked process if i do threads->exit(). But the script will still get stuck after a random number of forked processes.	[reply]
Re: problems with fork from thread by zentara (Cardinal) on Mar 01, 2012 at 19:37 UTC
I havn't run your code, but from the discussion, I suspect your problem may be a waitpid problem when you fork from the threads. I would try putting `$SIG{CHLD} = 'IGNORE';` [download] in both the main thread and the spawned threads. See Best way to kill a child process for more discussion. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply] [d/l]
Re^2: problems with fork from thread by Eliya (Vicar) on Mar 01, 2012 at 21:04 UTC
I don't think this would help, because (according to my tests) it's the child processes that hang in a FUTEX_WAIT (as nothing seems to be calling FUTEX_WAKE again any more): `7160 futex(0x808248, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 7162 futex(0x808248, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> 7164 futex(0x808248, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> ...` [download] (filtered `strace` output) Also, the results vary structurally (not only quantitatively) with the number of threads started, whether you run the script under `strace`, etc., so I suspect it's some race condition issue. In other words, a varying number of children simply don't terminate, so there are no corresponding SIGCHLD signals. Unfortunately, I don't know enough about Perl's thread implementation to be of any real help here... So, until someone more knowledgable in the area has found sufficient motivation and time to tackle those issues, I'd stick with the common recommendation to stay away from mixing threads with fork.	[reply] [d/l] [select]