threads->create hangs

duane_ellis2 has asked for the wisdom of the Perl Monks concerning the following question:

I have a problem on win32 (64bit win 7, perl v5.20.2, active state), where threads->create() hangs.

I am looking for debug pointers.

The code launches two processes: Process (1) a background process, and process(2) is GDB, the gdb target talks to the background process via a socket, I am creating an automated sw test environment, I need to control, interact and capture output (stdout) from the processes.

I'm using IPC::Open3() to create both processes processes, like this:

    # i,o,e are std in/out/err

    $p = open3( $i, $o, $e, @cmdline );
    $hr->{'alive'} = 1;
    $hr->{'pid'} = $p;
    $hr->{'stdout'}= $o;
    $hr->{'stdin'} = $i;
    $hr->{'stderr'} = $e;
[download]

To process the stdout, I create a reader thread like this:

 # The reader is given the hash reference, the 'o' 
 # indicates this is the 'stdout' reader 
 $hr->{'o_thread'} = threads->create( \&reader_thread, $hr, 'o' );
[download]

The above works quite well when I run one sub process. It locks up (the create call does not return) if I have two sub processes.

My only means to Debug is via print() - tells me that the call to threads->create() does not return, and print statement I put before threads->create() occurs, the after does not. And the print statement I put at the entry to the reader_thread never happens,

To be clear, this works a few times then does not work again.

Any suggestions how to dig deeper and figure out what I am doing wrong. Often, I insert print statements into various PM modules, but - threads are not perl, they are native code.

The reader thread is quite simple:

sub reader_thread {
    my ($hr,$who) = @_;
    
    my $c;
    my $h;
    my $r;
    my $q;

    threads->detach();

    # determine handle.
    if( $who eq 'o' ){
    $h = $hr->{'stdout'};
    } else {
    $h = $hr->{'stderr'};
    }
    # Get the output queue
    $q = $hr->{'Q'};

    while( 1 ){
    if( $hr->{'alive'} == 0 ){
        last;
    }
    # Try to read *ONE* byte
    $r = read( $h, $c, 1 );
    if( $r > 0 ){
        if( length($c) ){
        $q->enqueue($c);
        }
    } else {
        threads->yield();
    }
    }
}
[download]

Comment on threads->create hangs Select or Download Code

Replies are listed 'Best First'.
Re: threads->create hangs by BrowserUk (Patriarch) on Aug 15, 2015 at 00:33 UTC
Could you please post working code. Preferably a cut-down minimal version that demonstrates the problem. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice. I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!	[reply]
Re: threads->create hangs by ikegami (Patriarch) on Aug 17, 2015 at 01:48 UTC
Did you do anything to prevent `read` from blocking? You should get rid of that `yield`. If you need it, you're doing something really, really wrong.	[reply] [d/l] [select]
Re: threads->create hangs by anonymized user 468275 (Curate) on Aug 16, 2015 at 00:12 UTC
The concept of threads is to control concurrent use of the same code. If the two processes are intended to execute different code concurrently, you probably want to use fork instead, e.g.: `my $pid = fork; if (!$pid) { # execute background code exit; } else { # execute concurrent foreground code waitpid $pid, 0; } # execute non-concurrent foreground code` [download] One world, one people	[reply] [d/l]
Re^2: threads->create hangs (vs fork) by tye (Sage) on Aug 16, 2015 at 02:58 UTC
The concept of threads is to control concurrent use of the same code. If the two processes are intended to execute different code concurrently, you probably want to use fork instead No, the difference between threads and forking is that threads share the same memory while forked processes do not. Forked processes often run the same code and threads often run different code (though they can't run different programs, of course). The practical differences between sharing memory and not is that threads will usually have to use locks to prevent conflicting accesses to the same bits of memory but can also just store data in memory to later be used directly by another thread. While processes need to use IPC to send data back and forth and very often don't need to use locks. Note also that getting locking right can actually be a significant technical challenge when using threads, especially as the software gets older and more complex (so using separate processes is often considered "safer"). The original poster noted "I create a reader thread". Using fork to create "a reader" doesn't actually help because the new "reader process" would have to send the read data back to the parent via IPC which means the parent would have to read data from the reader process, which likely poses the same problem that prompted creating a new thread in the first place. Though, you could use non-blocking read or, to be even more efficient, 4-argument select to avoid creating a reader thread. Though that certainly adds some complexity as well, especially in Perl. And with a thread you can let Perl mostly deal with the locks for you so that is likely the simplest approach to code, anyway. - tye	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.
Re: threads->create hangs by anonymized user 468275 (Curate) on Aug 16, 2015 at 20:44 UTC
So, the story so far is that some responders avoid the question of the warning against using threads in threads. Conversely, an anonymous poster claims the warning can simply be removed. My position is that the correct use of Perl threads and forks is best derived from the pre-existing theory (described surprisingly well on Wikipedia). I have two critics Tye and browserUK who on the one hand make no comment about this warning but are clearly knowledgable about the Perl implementation. My challenge to anyone who wants to participate in the public interest is to engage with me in a competition to propose alternative and accurate documentation to threads that enables the reader to have a clear understanding of when to use threads versus forks. I don't set a time limit and I am starting my own efforts forthwith. Submissions, when ready should be linked in a reply to this topic. One world, one people	[reply]
Re^2: threads->create hangs by ikegami (Patriarch) on Aug 17, 2015 at 01:42 UTC
So, the story so far is that some responders avoid the question of the warning against using threads in threads. I asked p5p about that not long ago, and they very clearly said it's not a warning against using threads. It's a warning that threads aren't lightweight. If you're prepared to accept that, then use of threads is perfectly fine.	[reply]
Re^3: threads->create hangs by anonymized user 468275 (Curate) on Aug 17, 2015 at 09:08 UTC
Excellent, but then don't you think the documentation should more clearly present that view? If you had to ask, then it didn't do that for you originally, did it? One world, one people	[reply]
Re^2: threads->create hangs (warning) by tye (Sage) on Aug 17, 2015 at 01:29 UTC
You can read a lot about the motivations for the creation of that warning (certainly in the p5p archive but also other places). It has more to do with many people having a difficult time using threads than with any specific problems with Perl's implementations of threads. Well, except for the heavy per-thread memory cost that is specific to Perl's implementation (which is the part you misread as saying that Perl threads don't use OS threads). It is meant to discourage casual use of threads and has more social motivations than technical. Have fun fighting with p5p on what to replace it with. I really do wish you luck (sincerely), but I am not interested in participating in that. I try to stick to activities where my strengths and weaknesses are more beneficial. The warning should certainly be made less likely to mislead. - tye	[reply]
Re^3: threads->create hangs (warning) by anonymized user 468275 (Curate) on Aug 17, 2015 at 09:10 UTC
tye, many thanks for your sincere and informative response. One world, one people	[reply]
Re^2: threads->create hangs by Anonymous Monk on Aug 16, 2015 at 21:01 UTC
So, the story so far is that some responders avoid the question of the warning against using threads in threads. Conversely, an anonymous poster claims the warning can simply be removed. My position is Your position is irrelevant as its not based on information/knowledge but propaganda	[reply]
Re^2: threads->create hangs by SimonPratt (Friar) on Aug 17, 2015 at 12:01 UTC
I don't read it as a warning to not use threads. I read it as a warning that threads are not light-weight and people writing code using them are not likely to get much help. Through my own experience in writing threaded programs, I find the warning to be absolutely valid, although I suspect the number of people becoming proficient at using threads has grown somewhat since the warning was first written.	[reply]
Re^3: threads->create hangs by anonymized user 468275 (Curate) on Aug 17, 2015 at 14:53 UTC
Fair enough. Perhaps I am setting too high an expectation on the docs. The systems I worked on most recently have too trivial a usage of threads to be pushing the boundaries. So perhaps I should try creating a test set (destruction-testing ;)) for all the methods in the threads space - that might reveal whether there are specific problems that could be documented away - in my world there is such thing as a documentation problem i.e. what appears to be a bug isn't so long as it's documented properly - but for now I can't get past the fact that although some others clearly see it differently, I stare at those words and it's as if it is saying to me: 'abandon hope all ye who enter here!' - I can't remember anything else in perldoc that reads back at me anything even close to that. One world, one people	[reply]