No need to get ironic---my point was merely that threaded code is hard to get right, even for a good and experienced programmer. Your first example illustrated that well, and so does your fix. Here's a comparison of runtimes:

$ time perl threads_yours.pl 
real    7m13.312s
user    2m1.484s
sys     10m8.982s

$ time perl threads_mine.pl 
real    0m2.209s
user    0m6.028s
sys     0m0.604s

[download]

And here's a bit of the output:

$ uniq -c outfile | head
   2275 Thread 1
      1 ThreaThread 2
    454 Thread 2
      1 TThread 3
    909 Thread 3
      1 ThThread 5
    454 Thread 5
      1 Td 1
   1364 Thread 1
      1 Thread 1hread 2

[download]

This is a bog-standard Perl 5.10.1 on AMD64/i7 as it comes with Debian Squeeze¹. Either you overlooked yet another pitfall (I can't tell what it is---my point again: it looks deceptively simple but ain't) or the library implementation is buggy, either way its behavior is certainly more correct than my example's at the cost of being almost 200 times slower, but not quite correct yet.

As for the POE code, it took me indeed some 15 minutes to write:

#!/usr/bin/perl 
use strict;
use POE qw(Component::Client::HTTP);
use HTTP::Request;

my @names = qw/ zezima fred bill john jack /;
open my $fh, '>', 'outfile' or die $!;

sub start_req {
    my $name = shift @names or return;
    POE::Kernel->post(weeble => request => response =>
        HTTP::Request->new(
            GET => "http://rscript.org/lookup.php?type=track&time=6289
+9200&user=$name&skill=all"
        ), $name
    );
}

POE::Session->create(
    inline_states => {
        _start    => sub {
            POE::Component::Client::HTTP->spawn;
            start_req for(1 .. 5);
        },
        response  => sub { 
            my $name   = $_[ARG0]->[1];
            my $result = $_[ARG1]->[0]{_content};
            if($result =~ m/gain:Overall:\d+:(\d+)/isg) {
                print { $fh } "$name $1\n";
            } elsif($result =~ m/(ERROR)/isg) {
                print { $fh } "$name doesn't exist \n"
            } else {
                print { $fh } "$name 0\n";
            }
            start_req;
        },
    },
);
POE::Kernel->run;
close $fh;

[download]

I haven't used that component the last two years, so yeah, I was slow because I had to look up the defaults and how to pass the HTTP::Request object again. As long as an implementation will not trigger my pager by barfing at 4 in the morning and then look all innocent when I try and debug it, I think that's time well spent.

As for efficiency: no. The 500 do-almost-nothing threads in your code (mine didn't run long enough to register in top) need a resident set of 327 MB here (virtual size is slightly over 4 GB), I didn't try with the web scraper but I don't see how it could do any better. If I start 500 parallel requests in the POE version (well, my line here is 256 kbit on a sunny day during low tide ...), it takes 21 MB as opposed to 18 with five requests. Code-size-wise, all POE modules I have installed together are slightly over 40 kLOC including POD. I'll leave the comparison to Perl's thread code plus pthreads or whatever that builds on on your box to you. My user code is 36 non-empty lines (OK, I cheated you for two lines in the if/elsif block because that matches my style), yours is 42 so far.

¹ 5.14.2 on a newer kernel and a Phenom-II shows the same behavior, it only takes 260 MB but is even slower, I stopped it after over 20 minutes of CPU time

In reply to Re^8: Consumes memory then crashs by mbethke
in thread Consumes memory then crashs by allhellno

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.