rsh <defunct> processes appear when using fork and system calls

whatwhat has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I observe 'rsh <defunct>' processes appear when I run the perl code below. Now, I don't know if this is normal behavior, and even if it is abnormal behavior I am not certain it causes any problems. I am hoping you monks can tell me (1) if this is Normal/Abnormal behavior, (2) why these 'rsh <defunct>' processes occur, and (3) if it this could cause problems (in particular, could this lead to the parent losing track of the grandchildren)?

The goals and explanation of my code are the following: (1) Run 24 copies of a non-perl program on 24 remote nodes (computers). (2) When these 24 independent programs finish, the perl code recognizes this, does some analysis of the output, and then runs 24 more copies of the non-perl program on the 24 remote nodes. This process is repeated over-and-over.

Towards this end my perl code is set up in the following way: I do 24 forking commands (set in a loop), that make a system call that sends a 'rsh' command to the remote nodes, telling them to run the remote program. So my perl script is the 'parent', after the fork the system call is the 'child' and the rsh command to the remote node spawns a 'grandchild'. I use 'system' instead of 'exec' so that the child process waits for the grandchild to finish.

Here is the essence of my perl code (the original code is much longer):

#!/usr/bin/perl
use IO::File;
use POSIX ":sys_wait_h";

open(JUNKD,">test_rsh-commands.txt");
for($j=1;$j<=10;$j++) # Number of time to resubmit the 24 subprocesses
  {
# Phase 1: Setup phase to spawn jobs
&spawn_jobs(1,handle_child); 
for($i=1;$i<=24;$i++) # BEGIN: Number of subprocesses 
  {
  $proc = "sp" . "$i"; # remote node to run command on
  $cmd = "date"; # simple test command
  
  # Phase 2: Spawn the jobs
  &spawn_jobs(2,$proc,$cmd,1,2,3);
  } # END: loop
# Phase 3: Wait for the jobs to finish
&spawn_jobs(3,26);
close JUNKD;
  }

## BEGIN: Spwan children jobs on slave nodes ##
sub spawn_jobs
  {
  my @a=@_;
  my $phase,$i,$proc,$nt,$mc,$um,$sleep,$sub;

  $phase = $a[0];
  if($phase==1) { $sub = $a[1] }
  elsif($phase==2) { ($proc,$cmd,$nt,$mc,$um) = @a[1..5] }
  elsif($phase==3) { $sleep = $a[1] }
  
  if($phase==1) # Setup phase
    {
    # set up child signal handler
    $SIG{'CHLD'} = \&$sub;
    $|++;
    %fhlist;
    %fhlist2;
    %fhlist3;
    }
  elsif($phase==2) # Spawn the jobs phase
    {       
    # Create an anonymous file handle
    $pid = fork();
    if($pid < 0 or not defined $pid)
      {
      print LOG "$#-> Can't fork! Bad kernel!";
      close LOG;
      die "$#-> Can't fork! Bad kernel!";
      }
    elsif($pid == 0)
      {
      # child process 
      print JUNKD "/usr/bin/rsh $proc $cmd\n";
#      system("/usr/bin/rsh $proc $cmd");
# I'm commmenting out the above line, since not everyone
# has 24 remote nodes to run on.
#      system("$cmd");
      exec("$cmd");
      exit(0);
      }
    else
      {
      # Parent process, toss child file handle into the hash and move 
+on with
      # our lives.
      $fhlist{"$pid"} = $nt;
      $fhlist2{"$pid"} = $mc;
      $fhlist3{"$pid"} = $um;
      }
    }
  elsif($phase==3) # Wait till the children are done phase
    {
    while(1)   
      {
      @kl = keys(%fhlist);
      if($#kl >= 0)  
        {
        # mo' to do...
        sleep($sleep);
        }
      else
        {
        last;
        }
      }
    }
  }
### END: Spwan children jobs on slave nodes ##

sub handle_child 
  {
  # This gets called when a child dies... maybe more than one
  # died at the same time, so it's best to do this in a loop
  my $temp, $mcopy, $umbr, $nbias, $nmat;
  while(($dead_kid = waitpid(-1, WNOHANG)) > 0) 
    {
    $temp = $fhlist{"$dead_kid"}; # get the file descriptor back
    $mcopy = $fhlist2{"$dead_kid"};
    $umbr = $fhlist3{"$dead_kid"};
    delete($fhlist{"$dead_kid"});
    delete($fhlist2{"$dead_kid"});
    delete($fhlist3{"$dead_kid"});
    }
  }
[download]

Here is the evidence that 'rsh <defunct>' processes are occuring on the node where the parent is running (Please note the following when interpreting the data below: (A) The defunct processes appear only on the node where the parent perl script is running, (B) 'mubrex_mpi_biow' is the name of the perl script, and (C) for the sake of brevity this data is from the case when 13 subprocesses are run, NOT 24 as in the code above):

p243~/>ps -u user
  PID TTY          TIME CMD
10319 ?        00:00:00 tcsh
10320 ?        00:00:00 pbs_demux
10341 ?        00:00:00 439291.biobos.S
10367 ?        00:02:09 mubrex_mpi_biow
20933 ?        00:00:00 mubrex_mpi_biow
20934 ?        00:00:00 rsh
20935 ?        00:00:00 mubrex_mpi_biow
20936 ?        00:00:00 rsh
20937 ?        00:00:00 mubrex_mpi_biow
20938 ?        00:00:00 rsh
20939 ?        00:00:00 mubrex_mpi_biow
20940 ?        00:00:00 rsh
20941 ?        00:00:00 mubrex_mpi_biow
20942 ?        00:00:00 rsh
20944 ?        00:00:00 mubrex_mpi_biow
20946 ?        00:00:00 mubrex_mpi_biow
20947 ?        00:00:00 rsh
20948 ?        00:00:00 mubrex_mpi_biow
20949 ?        00:00:00 rsh
20950 ?        00:00:00 mubrex_mpi_biow
20951 ?        00:00:00 rsh
20952 ?        00:00:00 mubrex_mpi_biow
20953 ?        00:00:00 rsh
20954 ?        00:00:00 mubrex_mpi_biow
20955 ?        00:00:00 rsh
20956 ?        00:00:00 mubrex_mpi_biow
20958 ?        00:00:00 mubrex_mpi_biow
20945 ?        00:00:00 rsh
20957 ?        00:00:00 rsh
20959 ?        00:00:00 rsh
20960 ?        00:00:00 rsh <defunct>
20961 ?        00:00:00 rsh <defunct>
20962 ?        00:00:00 rsh <defunct>
20963 ?        00:00:00 tcsh
20964 ?        00:00:00 rsh <defunct>
20965 ?        00:00:00 rsh <defunct>
20968 ?        00:00:00 rsh <defunct>
20969 ?        00:00:00 rsh <defunct>
20972 ?        00:00:00 rsh <defunct>
20973 ?        00:00:00 rsh <defunct>
20974 ?        00:00:00 rsh <defunct>
20976 ?        00:00:00 rsh <defunct>
20978 ?        00:00:00 rsh <defunct>
20980 ?        00:00:00 rsh <defunct>
[download]

Thanks! Ed

Comment on rsh <defunct> processes appear when using fork and system calls Select or Download Code

Replies are listed 'Best First'.
Re: rsh <defunct> processes appear when using fork and system calls by Corion (Patriarch) on Aug 06, 2007 at 08:10 UTC
When Hell's Disk Is Full, Zombie Processes Shall Rise And Walk The Face Of The ... You need to wait for your children from time to time, so they don't become zombies. Basically, keep a list of the PIDs of your children, spawn them all and then wait until they're all finished. Or do it with a signal handler. See perlipc for more information.	[reply]
Re^2: rsh <defunct> processes appear when using fork and system calls by whatwhat (Novice) on Aug 06, 2007 at 13:42 UTC
But this is what I do with the 'waitpid' function. In the child signal handler: `while(($dead_kid = waitpid(-1, WNOHANG)) > 0)` [download] This is what is recommended in O'Reily. If this is incorrect could you give a short example? Also, I want to emphasize that these defunct processes do not accumulate. That is they are removed after I delete the child pids in the script. This seems to indicate that they are becoming defunct when the child makes the system call to issue rsh (?). And after the child finishes, these 'rsh <defunct>' processes disappear. Strange, No? Thanks! Ed	[reply] [d/l]
Re: rsh <defunct> processes appear when using fork and system calls by cdarke (Prior) on Aug 06, 2007 at 14:52 UTC
I suspect the problem is with mixing CHLD signal handling with system. You might try explicitly calling waitpid in the parent rather than using a signal handler. Why does the child have to wait for the grandchild to finish? You could set SIGHUP to IGNORE, which should avoid killing the grandchild when the child exits, and use exec (or use nohup, for the lazy).	[reply]
Re^2: rsh <defunct> processes appear when using fork and system calls by whatwhat (Novice) on Aug 07, 2007 at 08:36 UTC
I'll try rewriting my code with waitpid in the parent. Although I have experience with perl, I am a novice in forking processes, etc. So i don't know if this is the best, in my case 'best' means stability, code for doing what I want to do. My reasoning for having the child wait for the grandchild is two-fold. One the grandchild is a non-perl program that runs for ~1 minute. So if I didn't use 'system', which forks and waits for its child to finish, and used 'exec' instead (which as I understand it just executes the command, and does not wait for its child to finish) then I would put multiple jobs (each needing heavy CPU usage) running on one node, instead of running them serially. This is extremelly undesirable outcome for me. The second reason I have the child wait for the grandchild is that I need to analyze the output of the 24 non-perl programs when they are finished, the only way I know how to do that is in side the child signal reaper. Please let me know if my assumptions above regarding 'exec' are incorrect. If I could use your suggestion and get rid of the defunct rsh processes, while still meeting the criteria above, please let me know. Thanks!	[reply]
Re^3: rsh <defunct> processes appear when using fork and system calls by cdarke (Prior) on Aug 07, 2007 at 12:53 UTC
exec is a little different to how you describe, although the effect may be similar. The exec command does a rather peculiar thing: it replaces the current program with a different one in the same process. There is no return from a successful exec, since the original code is lost. Some things do survive an exec, the PID, open file handles, DEFAULT or IGNORE signal disposition, and ENVironment variables, to mention a few. You imply you are doing some processing after the call to system, yet looking at your code you appear to be doing an exit(0). If it got to that point then the exec did not work, so a die (showing $!) might be better. In the parent it is perfectly normal to call waitpid, there is no need to place it in a CHLD signal handler. Using an argument of -1 (no need for WNOHANG) will wait on the next child to finish, and return its PID. For example (untested): `while (keys %fhlist) { my $pid = waitpid(-1, 0); delete $fhlist{$pid}; }` [download] By the way, if you call waitpid (or its sister wait) for each of your children then you avoid zombies.	[reply] [d/l]
Re: rsh <defunct> processes appear when using fork and system calls by Argel (Prior) on Aug 07, 2007 at 20:41 UTC
If your system supports autoreaping of zombies you might try enabling that via the following: `$SIG{CHLD} = 'IGNORE';` [download] However, I am not sure if the calls to system you are making will interfere with this or not. I think if you switched to exec it is more likely to work.	[reply] [d/l]
Re: rsh <defunct> processes appear when using fork and system calls by ikegami (Patriarch) on Aug 07, 2007 at 20:47 UTC
Why do you have `spawn_jobs 1,`, `spawn_jobs 2,` and `spawn_jobs 3,` instead of three functions? It makes it hard to read your code and therefore hard to debug your code.	[reply] [d/l] [select]
Re^2: rsh <defunct> processes appear when using fork and system calls by whatwhat (Novice) on Aug 08, 2007 at 11:36 UTC
For the sake of generality. In the full code (not shown) I have 4 different types of commands I need to spawn. If I didn't have the forking as a subroutine I would have to repeat most of the subroutine code 4 different times, making the code longer, and more difficult to modify (eg if I needed to change code in one of the forking sections I probably would need to change it in the 3 others).	[reply]