expresspotato has asked for the wisdom of the Perl Monks concerning the following question:

Hello, So I'm running a perl script that actively scans a directory, checks for a small flat file, imports its items, then runs a separate perl process using a thread then a system call. For some reason, and one point or another it completely locks up. No error messages, no output, nothing. Just seemingly doesn't continue with its usual "..." manager. Is there a limit to the number of active threads? Or a limit on how many perls can be running at once? Some threads run for hours, while some for mere minutes. All in all it seems to work well, for an hour or two that is. Any help would be appreciated my Monks! The code's attached too. 

#! /usr/bin/perl

use threads;

print "Starting Fork...\n";
print "Might as well cleanup first...\n";
print "Ending All Python...\n";
system("killall python");
print "--> Cleaning System\n";
system("rm -Rf 100*");

sleep(1); print "Making sure system is ok...\n";
system("mkdir ./in_use ./use ./finished ./finished_already ./finished_
+downloading ./already_done ./acc");
system("chmod 777 ./in_use ./use ./finished ./finished_already ./finis
+hed_downloading ./already_done ./acc");

print "\nFinished Cleaning... Startup! \n";
sleep(1);

$thr1 = threads->new(\&manager);

while (1){sleep(1);}

sub initiator(){

    print "Found (@_)\n";

    my @in_items = @_;
    
    print "System...\n";
    system("perl silly_fork_handler.pl @_");
    print "System Ended!\n";
}

sub nothing(){
}

sub manager(){
    $|++;
    while(1){
        print ".";
        opendir(DIR,"./acc"); @list = readdir(DIR); close(DIR);
        foreach (@list){ if ($_ =~ ".dat"){ @new_list[$q] = $_; $q++; 
+}}
        #print @new_list;

        foreach (@new_list){
            open(READER,"./acc/$_"); @items = <READER>; close(READER);
            #print @items;
        
            chomp($_);
            @list5 = $_;
            $thr = threads->new(\&initiator, @list5);
            $thr = threads->detach;
        
            $line = $_; chomp($line);
            #cp("./acc/$line","./$line/FORKED.dat");
        }
        
    opendir(DIR,"./finished"); @list2 = readdir(DIR); close(DIR);
    
    foreach (@list2){
        $line = $_; chomp($line);
        if (($line ne ".") && ($line ne "..")){
            $output = `tail -n 1 ./$line/WATCH.dat`; chomp($output);
            if ($output =~ "ERROR:"){system("cp ./$line/WATCH.dat ./tm
+p.dat");}
                unless ($line eq ""){
                    system ("rm -rf ./$line");
                    system ("rm -rf ./finished/$line");
                }
            if ($output =~ "ERROR:"){mkdir("./$line"); system("cp ./tm
+p.dat ./$line/WATCH.dat");}
            print "\nRemoved buffered torrent ($line)\n";
        }
    }
    
    undef @list; undef @new_list; undef @list2;
    undef @items; undef $line;
    $q = 0;
    sleep(2);
    }
}

[download]

Replies are listed 'Best First'.
Re: Completely Random Stall
by BrowserUk (Patriarch) on Jun 12, 2008 at 13:06 UTC

    I suspect, but cannot confirm that if you were to add strict and warnings and clean up all the messages they produce, then you would probably go a long way to fixing your problem.

    One definite error is that you are never cleaning up your threads, because this code make no sense whatsoever: 

                $thr = threads->new(\&initiator, @list5);
            $thr = threads->detach;

    [download]

    That should probably be, 

                $thr = threads->new(\&initiator, @list5);
            $thr->detach;

    [download]

    Which means none of your threads are being discarded when they complete, so you will eventually run of of threads. How quickly may depend upon the limits your of your OS, or of some internal details of the Perl implementation.

    But fixing that may not itself be sufficient to fix your problem because $thr, along most everything else, is a global var. This may be benign, but since I never program that way, I have no notion as to the implications, so it gives me cause for concern.

    Also, as presented, there is no purpose in running manager() (once) as a thread, and then having your main thread go into a do nothing loop. You can replace these two lines: 

    $thr1 = threads->new(\&manager);

while (1){sleep(1);}

    [download]

    with: 

    manager();

    [download]

    And your program will run exactly as it does now, but with one less thread doing nothing.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      So how could I detach the threads? Can I do so within the thread it self by calling threads->detach(); from within the thread? ? Also I honestly think you're right, hundred or so threads down the line its just stalling (but the previously running threads ' sub-processes are still running).
[reply]
[d/l]
        So how could I detach the threads?

        I showed you exactly how to do it above? Change threads->detach; to $thr->detach;.

        detach() is an instance method. That is, you need to call it upon the thread handle you wish to detach. You are currently calling it as a class method, which makes no sense, and if warnings were enabled, would (probably) raise an error.

        Does your OS have a tool that will show you how many threads are running when the stall occurs? Also, are your child processes being cleaned up or are you creating lots of zombies?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
Re: Completely Random Stall
by chrism01 (Friar) on Jun 13, 2008 at 07:02 UTC
    I'd like to second BrowserUK's advice, esp re use strict, warnings.
    Also, replace your system() calls with the built-in Perl equivalents, which will give you better error handling and remove the process-creation overhead you're getting.
    As well, check for errors in opendir() : http://perldoc.perl.org/perlopentut.html.
    BTW, forking != threading.
[reply]
      Hi Monks! I'm still having the same problems... A few hundred threads in, the script decides to stop working. How should I be cleaning up after the thread has finished? Any help is appreciated. 
      #! /usr/bin/perl

use threads;

$|++;
print "Running...\n";

while(1){
    opendir(DIR2,"./bgthread/process"); @list3 = readdir(DIR2); close(
+DIR2);    
    $line = "";    @items = "";
    foreach (@list3){
        $line = $_; chomp($line);
        if (($line ne ".") && ($line ne "..")){
            ($buffer,$do,$torrent) = split(/\-\.\-/,$line);
            print "Sending ($line)\n";
            $thr = threads->new(\&do_thread);
            $thr->detach;
        }
    }

    if (-e "./bgthread_refresh"){
        exit;
    }
    sleep(1);
    #system("sleep 0.5");
    print ":";
}

sub do_thread(){
    system("perl bgthread_processor.pl $line");
    #goto &nothing;
}


sub nothing(){
    #Do nothing...    
    undef $line;
}

      [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom