I have an application that tails multiple files using File::Tail and reports realtime values on each. Each File::Tail object monitors a different file, so I figure the only way of implementing this is to use threads.

From the perldoc page for threads, I see I should use ithreads, since it has less problems. I also read this post, linked from the tutorial page. Since the former doesn't have a lot of tips other than to use ithreads and the latter is a couple years old, anybody have thoughts/updates/etc.? Would you say threads is now ready for production systems? I know there were a lot of problems with threads prior to v. 5.6; looks like these issues have been addressed?

-- Burvil

Replies are listed 'Best First'.
Re: Stability of threads in perl 5.8?
by erroneousBollock (Curate) on Nov 05, 2007 at 04:23 UTC
    I use iThreads in productions systems, no ongoing problems (running 2-5 months, ~900k requests)... YMMV.

    You'll need to:

    • understand the non-shared cloning behaviour
    • understand the shared-var clobbering behaviour
    • understand where the magic will/won't hold up

    I mostly use Thread::Queue to communicate between a reusable pool of threads. I try not to use shared variables (so I don't have to reason about what is/isn't shared). I make sure not to allocate *anything* I don't have to in the "parent" thread.

    There certainly are still some problems. There's no other option on Win32; fork() there is an emulation using iThreads which breaks other things (so you shouldn't use it).

    I'm sure others will have war stories... I've got what I needed to done without too many problems... mostly working around cloning and file-handle DESTROY issues.

    -David

Re: Stability of threads in perl 5.8?
by fenLisesi (Priest) on Nov 05, 2007 at 09:06 UTC
Re: Stability of threads in perl 5.8?
by zentara (Cardinal) on Nov 05, 2007 at 12:58 UTC
    You really only need threads if you want to share data between threads in realtime. If not, forking is probably a better choice. Use a fork manager module to fork off a process for each file to be tailed.

    If you need to constantly monitor the file tail results, you could use threads, but there are alot of caveats to watch for when using threads. Many times, threads need to be created before any other object code is called(due to non-threadsafe objects), so you may not be able to just use threads any way you want. Threads work well if properly setup for the situation, but you show use the SuperSearch box here and search for all the previous posts on threads. See Why use threads over processes, or why use processes over threads?


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
      forking is probably a better choice

      But only if you are on a platform that supports a native fork. Otherwise you;re just using threads but without the control and a whole different set of caveats in addition to those of threads.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Stability of threads in perl 5.8?
by BrowserUk (Patriarch) on Nov 05, 2007 at 16:09 UTC

    For simplicity, does it get much easier? Just supply the name of the directory containing the files and you're off. ^C to end it.

    #! perl -slw use strict; use threads; use threads::shared; use File::Tail; my $quit :shared = 0; $SIG{ INT } = sub{ $quit = 1; warn 'Quiting'; }; sub thread { my $tid = threads->self->tid; my $name = shift; my $file = File::Tail->new( $name ); while( not $quit and defined( my $line = $file->read ) ) { print "$tid ($name) '$line'"; } } $_->join for map{ threads->create( \&thread, $_ ) } glob $ARGV[ 0 ];

    For stability, make sure you use the latest version of Perl (5.8.8), threads, and threads::shared.

    If you test your solution and are happy with it before you put it into production, then it should be stable. As with pretty much any feature of perl, it is possible you will encounter a bug once in production, but if you've stayed within the limitations of the features you use, and tested thoroughly, you'd be unlucky to do so.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Stability of threads in perl 5.8?
by jplindstrom (Monsignor) on Nov 05, 2007 at 14:26 UTC
    I've done that exact thing single-threaded with File::Tail and, I'm pretty certain it was, IO::Select. More robust and less complex than threads for this particular problem I'd wager.

    However, if you do use threads and need to process the content in one single place, consider isolating each File::Tail in its own thread, and (as others have mentioned) use Thread::Queue to pass values to the thread processing the data.

    /J

Re: Stability of threads in perl 5.8?
by chrism01 (Friar) on Nov 13, 2007 at 04:50 UTC
    Allowing for the caveats expressed above eg by eb, threads in 5.8 seem fine to me.
    I wrote a threaded daemon for an ISP for tracking user's packet usage via netflow/RADIUS which runs 24/7 and it's been running fine for 1 or 2 yrs iirc (I've moved on a while ago).
    Just need to be careful and I'd also recommend having a stable num of thrs and re-using them, rather than creating/destroying thrs to be safe.

    Cheers
    Chris