in reply to Re^4: Problem with using threads with modules.
in thread Problem with using threads with modules.

Rather than messing around with IO::String or in-memory files and then needing to use substr to try and extract the new bits of the log etc., I think I have an alternative that will achieve your aim.

The basic idea is to tie a filehandle to a Thread::Queue object, which can be shared across threads. You tie a filehandle and then pass this into each of your Telnet threads and they pass it to their respective Net::Telnet objects for logging.

Your main thread can retrieve the thread::Queue handle from the tie and the loop over reading from it and displaying what it gets into the appropriate window(s). It can also, output the lines to the permeanent log file(s) within the same loop.

This 'loop' could also be a Tk event callback.

I hope the following (crude) code will demonstrate the idea.

#! perl -slw use strict; use threads qw[ yield ]; use Thread::Queue; package Tie::Glob::Queue; sub TIEHANDLE { my $class = shift; my $self = new Thread::Queue; return bless \$self, $class; } sub PRINT { my $Q = ${ shift() }; $Q->enqueue( join( $, || '', @_ ) ); } 1; package main; our $TELNETS ||= 10; my $threads : shared = 0; sub telnets{ { lock $threads; $threads++ } my $io = shift; for( 1 .. 5 ) { print $io threads->tid(), " : $_"; select undef, undef, undef, rand 1; } { lock $threads; $threads--; } } my $Q = ${ tie *LOG, 'Tie::Glob::Queue' }; my @threads = map{ threads->create( \&telnets, \*LOG )->detach } 1 .. +$TELNETS; yield while $threads < $TELNETS; while( $threads ) { if( $Q->pending ) { print $Q->dequeue; } else{ yield; } }

The tie package will need filling out (probably by inheriting from Tie::StdHandle) and the telnets() sub is just a placeholder but the basic methodology is there.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

Replies are listed 'Best First'.
Re^6: Problem with using threads with modules.
by tele2mag (Initiate) on Jun 28, 2004 at 14:27 UTC

    I have looked over your code during the weekend. I understand the idea but not all the syntax though.
    In our $TELNETS ||= 10;, what does " || " do?
    And this: select undef, undef, undef, rand 1; ?
    In $Q->enqueue( join( $, || '', @_ ) );, $ stands for the joining character and @_ is the arguments. But what is " || " and " '' " doing there?
    I'm not lacy, just couldn't find that type of syntax in my Perl litterature. I have Perl CD bookshelf (E-book) and Perl-power (hardcopy) amongst other.

    Other than that, it looks good. I have to inherit a skeleton class IO::Handle but leave the major methods blank. Hope telnet-socket only require writing to handle or this whole thing will not work easilly.
    It will not work as good if telnet-socket is writing in char-by-char fashion instead of line-by-line. I can check that out later.

    This is the behaviour I had in mind for the code your supplied:
    I use the main (GUI) thread to create a tie to the thread::queue as you have done.
    Main-thread starts a new telnet-thread and passes that tied queue. Telnet-thread creates a socket with logging to tied-filehandle. Then command-execution to the telnet-socket starts (and later ends) while logging occur.
    Main-thread then starts a Update-thread that uses the tied-handle to save logging to file and calls a callback function that output the logging to the right GUI window.
    For that to work, all the telnet threads must print tagged-messages to tied-handle so I can separate them in my Update()-thread and output them to the right file and window.

    So my program will have.

    • a Main-thread that handles GUI and user input
    • a Update-thread that updates GUI-window(s) and file(s) with telnet-traffic.
    • 1..n, Telnet-threads that connects and executes commands on various hardware devices.

    Main-thread may start another telnet-thread when user gives some appropriate GUI command. (So they don't start at the same time). The number of telnet-threads is limited by the number of hardware access points.

    This is not so different to your behaviour description and it looks promising. By the way, I got the message "A thread exited while 2 threads were running." while I ran your code. It was not an error so I was not to bad.
    Thanks, for redirecting me. Hope it works out for me now :)

      In our $TELNETS ||= 10;, what does " || " do?

      See C-style-Logical-Or

      || means 'or'. It is the same as the keyword or except it has a higher priority.

      ||= as in  $var ||= constant; is the same as  $var = ( $var || constant );. It is used (by me anyway) to give a variable a default value if that variable has not already been given a value.

      Caveat: If the value of $var is either 0 or '' (null string) or undef, then $var ||= 10; will overwrite that value with 10. So if 0, '' or undef is a legitimate value for $var, the ||= method of applying a default value is not useful.

      The idea is that using P:\test>perl -s 369260.pl -TELNETS=3 allows me to choose how many threads get started, but if I omit it, it defaults to starting 10. (Belatedly, there is a patch that allows a syntax //= that is somewhat more useful, but as of 5.8.3 this doesn;t seem to have made it into the mainstream yet.)

      Why not use Getopt::slighly_shorter_than_long_but_longer_than_short_and_not_quite_as_general_as_general?

      Why are there so many? Simple. Each has it's own set of limitations--along with variations on command line syntax, usage syntax, defaults syntax, etc. etc. etc. I tried several when I first started Perl and got thoroughly confused and frustrated by them. I hate (vermently) the --some_grossly_long_option(colon | is it =? | is space?)value syntax.

      Besides which, the built-in -s syntax ( See perlrun for details) is adaquate for most everything I need, is intsalled everywhere, requires little by way of learning curve, doesn't require 70KB of code.

      It's a personal preference:)


      And this: select undef, undef, undef, rand 1; ?

      See select. You can probably ignore most of it except the last 4 or 5 lines.

      Essentially, select undef, undef, undef, 0.1; is the same as sleep 0.1;, except that sleep doesn't permit sub 1-second delays.

      It effectively suspends the thread of execution for 0.1 seconds. You can also get a sub 1-second delay by installing Time::HiRes and allowing it's sleep to override the standard sleep built-in. Or if you use Win32, you can use Win32::Sleep( $delay ) that takes it argument in millseconds rather than seconds. The reason for using select in this way is that it wouldn't require you to install anything to try the program and it is cross-platform.

      rand 1; returns a real number between 0 and 1 (eg. 0.551361083984375 ), so the while thingselect undef, undef, undef, rand 1; basically says sleep for a random amount of time between 0 and 1 second.

      The reason for using it in my test app. is to simulate a multi-threaded telnet application by giving me fairly high-speed, randomly distributed activity amongst the threads, allowing me to test the tie-Queue, that was the main purpose of the test worked.


      In $Q->enqueue( join( $, || '', @_ ) );, $ stands for the joining character and @_ is the arguments. But what is " || " and " '' " doing there?

      See $, (you'll have search for it - "$OUTPUT_FIELD_SEPARATOR".

      join( $, || '', @_ ) simple says, use the value of $, as the separtor when joining the values in @_ together into a string, BUT, if $, is undefined, then use '' (the null string) instead.

      Why?

      When I used $, my thinking was that it would allow the user of the module (me) to set $, to whatever my preference for a separator was (globally) and it would be used. So if I wanted to set $, = ', '; and get a comma sperated string back I could. $, is normally set to '' anyway, but when I ran my test (with just join( $,, @_ )I got:

      I wasn't (and still am not) sure why this is so, but I wasn;t interested in solving that problem at that point in time, so I added || '' which 'fixed' the problem giving me the output I expected:

      I never went back and fixed it properly. It was only a demo after all.

      I'm not lacy, just couldn't find that type of syntax in my Perl litterature. I have Perl CD bookshelf (E-book) and Perl-power (hardcopy) amongst other.

      I also have the "Perl CD Bookshelf (version 3)", and it it is very good for looking up good starting points for many things. The one thing it lacks is a basic syntax description. For that the (in)famous perdoc, or better(IMO) the html-ized perl pods is the only source that I know of. As a reference work it is very complete, but it does take a considerable while to find your way around when you first start. Knowing where to look, or even what to look for is the basic problem, and there is an aweful lot of it. Perldoc.com is the best inline source I know of.


      Other than that, it looks good. I have to inherit a skeleton class IO::Handle but leave the major methods blank. Hope telnet-socket only require writing to handle or this whole thing will not work easilly. It will not work as good if telnet-socket is writing in char-by-char fashion instead of line-by-line. I can check that out later.

      The tie package I included was the bare minimum to test the idea. To complete it, you will need to read Tie::Handle and Perltie. In reality, there is surprisingly little more to add if you follow the advice therein and make your package inherit from Tie::Handle. You would need to add a WRITE method only I think.

      There was little purpose in my continuing to develop the idea beyond proving that it would work as I don't currently have a use for it, b) you may have already decided upon your course of action and never come back to see what I had posted. c) It would only be truely useful for those (probably rare) occasions where you need to pass a filehandle into another module for it to write to, but want to intercept the output--and then only if you are doing this in combination with threads.

      Yours is the first application I have seen that would need this. 1 apple hardly makes an orchard:). Maybe as threads become more used, it would be worth while making a cpan module of this, Time will tell.

      This is the behaviour I had in mind for the code your supplied: ...
      • First problem I see there is the Update thread and a call-back function.

        If the Update thread calls $Q-dequeue first, then the callbacl function will not be able to retrieve that data from the queue. And vice-versa. Once you unstacked data, you can't unstack it again.

      • The second is that it seems unnecessary to force all the output from several threads through a single queue, only to have to separate them again.

        When I wrote the test code, it wasn't clear to me from your description whether you wanted to keep the logging separate or not. As you appear to want to keep it separate, both in the window you want to display it, and in separate log files, why not just create a separate tied handleQ for each client?

        This removes the need for using a separate Update thread, and the dequeuing conflict that presents. Each client logs to a separate tied handleQ. Then the callback function for the window resposible for displaying that logging, dequeues the data from the appropriate Q and both displays it AND writes it to the appropriate logfile.

      I don't know enough about Tk to easily write a demo for this. Sorry.

      I've played with it a couple of times, starting with snippets posted by others, but I found the documentation incredibly difficult to use. It is in a zillion little pieces, with virtually no actual code somaples and spread all over the disk in chucks that are 90% repetitious headers and footers. In the few small tries I had with it, it was incredibly slow and clumsy. I had a much slower machine back then.

      Assuming, based on my knowledge of other windowing systems, that each Tk window receives some event to indicate that the window needs to be redrawn--lets call it the Update event--then the following (very) psuedo-code may clarify the idea.

      ### THIS IS PSEUDO CODE ONLY. IT WILL NOT RUN. IT WILL REQUIRE MUCH WO +RK. #! perl -w use strict; use threads; use threads::shared; use Thread::Queue; our $TELNETS = 10; ## Pre-create the queues that will be tied. our @queues : shared = map{ Threads::Queue->new } 1 .. $TELNETS; my $done : shared = 0; sub telnet{ my( $Qno ) = @_; require Net::Telnet; require Tie::Glob::Queue; open REALFILE, '>>', "LOGS/telnet.$Qno" or die $!; ## Pass in the Q handle for this tied handleQ tie *LOG, 'Tie::Glob::Queue', \$queue[ $Qno ]; my $telnet = Net::Telnet->new( ..., log_to => \*LOG, ... ); ... return; } ## Start the threads, pass the Q nos. ( 1 .. 10 maps to 0 .. 9 etc.) my @threads = map{ threads->new{ \&telnet, $_ -1 }-detach } 1 .. $TELN +ETS; ## Require (NOT use) the Tk stuff AFTER creating the threads. require Tk; ... sub update { my( $Qno, *LOGFILE ) = @_; while( !$done ) { ## Each time we are called, read the new data from the queue while( my $logdata = $queues[ $Qno ]->dequeue ) { ## And add it to the window $listbox->adddata( $logdata ); ## and the logfile print LOGFILE $logdata; } } } my $main = Tk->new(...); my @loggers = map{ $main->ListBox( update_callback => [ \&update, Qno => $_, LOGFILE = *LOGFILE ] ) } 1 .. $TELNETS; ## run the program MainLoop;

      That is VERY crude, but it outlines the basic steps I envisage. I hope that it helps some.

      Thanks, for redirecting me. Hope it works out for me now :)

      YW. If the produce of my deseased mind help someone out that's great. Now all I need is someone prepared to pay me for it:)


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

        This thread is running real deep (Re^8) :) Below a few comments to your post.

        Why are there so many? Simple. Each has it's own set of limitations--along with variations on command line syntax, usage syntax, defaults syntax, etc. etc. etc. I tried several when I first started Perl and got thoroughly confused and frustrated by them. I hate (vermently) the --some_grossly_long_option(colon | is it =? | is space?)value syntax.
        Well, I have to agree, but only to a certain level. To many variation confuses people and makes Perl only for the real experts to share code. Yours was not to bad, I have aldready adapted to that, but the problem is there. That's why Perlmonks have the section "Obfuscated Code" as I understand it :D

        There was little purpose in my continuing to develop the idea beyond proving that it would work as I don't currently have a use for it, b) you may have already decided upon your course of action and never come back to see what I had posted. c) It would only be truely useful for those (probably rare) occasions where you need to pass a filehandle into another module for it to write to, but want to intercept the output--and then only if you are doing this in combination with threads.
        To b): I allways return to follow up my posts or to help someone else for that matter.
        To c): Yes, it is a rare special case, but only for Perl I think. Network applications now uses more and more threads and to be able to monitor or log the traffic is useful. Perl is more restrictive than other languages like Java on sharing data amongst treads.

        If the Update thread calls $Q-dequeue first, then the callbacl function will not be able to retrieve that data from the queue. And vice-versa. Once you unstacked data, you can't unstack it again.
        But that was not my intension. Update()-thread were the only one who got to $Q->dequeue and then supply callback function with the data. But your pseudocode looks better so I leave that solution for the moment. (as usual when you convince me of a better way) ;)

        Assuming, based on my knowledge of other windowing systems, that each Tk window receives some event to indicate that the window needs to be redrawn--lets call it the Update event--then the following (very) psuedo-code may clarify the idea.
        In "Perl CD bookshelf"->Tk Chapter 15.2 there is a list of legal event types. See here: The closest is Expose but not the one I want. A general redraw-event may not be access at user level. Some investigate is needed to confirm that.
        But I don't know that it is going to work anyway. If the User just observe the window and don't interact with it, no redraw event will occur. You should be able to only observe the progress without taking any action that will generate the event.

        In the last line of your post, it started with YW, did not get it. I believe that it is a shortening or maybe slang of something. Enlighten me please :)
        I have put this program on hold for the moment until I have figured out how to do it, so I'm not in a hurry.
        Threads and Perl may not go hand-in-hand but I choose it because of its powerful text-handling capabilites.