Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re^9: Forks, Pipes and Exec (file descriptors)

by diabelek (Beadle)
on Nov 08, 2008 at 04:12 UTC ( #722354=note: print w/replies, xml ) Need Help??

in reply to Re^8: Forks, Pipes and Exec (file descriptors)
in thread Forks, Pipes and Exec

I can definitely change the buffering on the other application and that helps a bit.

As for the non-blocking, I thought non-blocking allowed a read to complete even when there was no data. If there was no data, I thought either EOF or EAGAIN or something was set. The example above can be changed to this to get it working in linux and loop as fast as possible.

use IO::Handle; use Timer::HiRes qw(usleep); print "before the pipe\n"; sub getpipe { my $hash2 = shift; print( "starting " . time() . "\n" ); $hash2->{pid} = open( $hash2->{stdout}, "/usr/bin/perl |") o +r die; $hash2->{buffer} = ""; $hash2->{stdout}->blocking(0); sleep( 1 ); return $hash2->{pid}; } my $hash = {}; getpipe($hash); my $fh = $hash->{stdout}; print( "pipe created\n" ); my $count = 0; while( 1 ) { my $data = <$fh>; if( $data != "" ) { print "got from pipe: $data"; } print( "." ); usleep( 10000 ); } print( "done\n" );

I will try to post the script externally and provide a link so you understand what I'm trying to do a little better. To answer a couple of your questions about threads/processes, I'm really using both. I start the thread/process by calling fork. So in Linux, I understand that I'm using the true fork command. In Windows, I'm using the pseudo fork that is really thread when you get to the bottom of things. Which one I use doesn't matter. I just need a separate branch of execution so the parent script can continue doing its normal thing while my branch of execution handles the external processes. Thats probably an ignorant statement but that's why I'm posting :)

I've found another perl script that sounds like what I'm doing: logtail. I believe the basic idea is the same. I need to review the code more to see what they have done.

Replies are listed 'Best First'.
Re^10: Forks, Pipes and Exec (file descriptors)
by BrowserUk (Patriarch) on Nov 08, 2008 at 15:12 UTC
    I've found another perl script that sounds like what I'm doing: logtail.

    Here's a simplified and somewhat crude equivalent that uses threads and should run anywhere you have a threaded-perl and a tail command. Maybe it'll be useful to you.

    #! perl -slw use strict; use threads ( stack_size => 4096 ); use threads::shared; use Thread::Queue; $|++; our $VERBOSE :shared; our $REMOTE :shared; my $stop :shared = 0; ## Set true to terminate threads my @logs = map glob, @ARGV; ## expand wildcards my $Q = new Thread::Queue; ## One trhead per log file threads->create( \&tail, $Q, $_, 1 )->detach for @logs; my $remote; ## Remote watcher socket if( $REMOTE ) { require IO::Socket; $remote = IO::Socket::INET->new( $REMOTE ) or warn "Couldn't connect to $REMOTE : $!, $^E"; print $remote "Hi there, Got your ears on?"; } ## Thread to monitor the Q, print locally and/or forward to remote my $relay = async { for( 1 .. @logs ) { ## Waits for all tals to terminate while( my $line = $Q->dequeue ){ chomp $line; print $line if $VERBOSE; print $remote $line if $remote; } } }; ## Local command loop while( <STDIN> ) { my( $command, $value ) = split; if( $command =~ m[^(END|QUIT)]i ) { $stop = 1; warn "Quiting...\n"; $relay->join; exit 0; } if( $command =~ m[^VERBOSE]i ) { $VERBOSE = $value; print "VERBOSE set to $value"; } elsif( $command eq 'qs' ) { print $Q->pending; } else { print "Unrecognised command: $command"; } } ## Tail threads sub tail { print threads->tid, ' : ', threads->self->get_stack_size; my( $Q, $path, $seconds ) = @_; $seconds = 1 unless $seconds; my $pid = open my $log, "tail -Fs $seconds $path |" or die $@; printf "Thread %d Following $path\n", threads->tid; $Q->enqueue( $_ ) while not $stop and defined( $_ = <$log> ); kill 3, $pid; close $log; $Q->enqueue( undef ); }

    It happily follows 100 logs, simultaneously logging them local and transmitting them to a remote watcher. You can enter a few commands at the local console to turn local verbose on and off, monitor the size of the queue, and quit:

    c:\test> -REMOTE=localhost:35007 log\log*.txt Thread 2 Following log\log0002.txt Thread 1 Following log\log0001.txt Thread 3 Following log\log0003.txt Thread 4 Following log\log0004.txt Thread 5 Following log\log0005.txt Thread 6 Following log\log0006.txt Thread 7 Following log\log0007.txt Thread 8 Following log\log0008.txt Thread 9 Following log\log0009.txt Thread 10 Following log\log0010.txt verbose 1 VERBOSE set to 1 Sat Nov 8 15:09:45.575 2008 : Message from log 6 Sat Nov 8 15:09:45.684 2008 : Message from log 0 Sat Nov 8 15:09:45.701 2008 : Message from log 3 Sat Nov 8 15:09:45.794 2008 : Message from log 3 ... ver ... Sat Nov 8 15:09:48.279 2008 : Message from log 9 ... bose 0 ... Sat Nov 8 15:09:48.309 2008 : Message from log 1 Sat Nov 8 15:09:48.419 2008 : Message from log 9 VERBOSE set to 0 qs 0 quit Quiting...

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      I've switched to using sockets and threads but I've run into an interesting problem. On Windows only (shocker), I am unable to start a thread inside of another thread. I'm unclear on why it stops since depending on where I place the thread->new() it works or doesn't work. I've posted the module code here.

      You'll notice in ToolHandler the while loops grabs a command and executes it and then tries to read the queue for the other processes that are running the IO tool. If I place the thread->new() before the _CommandHandler call, it fails to start the new thread and just hangs indefintely.

      I'm working on putting together a simple version of the module as a script to see if simplifying it changes anything but if there is any advice on the module it would be appreciated.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://722354]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (1)
As of 2023-09-21 21:01 GMT
Find Nodes?
    Voting Booth?

    No recent polls found