in reply to Re^7: Consumes memory then crashs
in thread Consumes memory then crashs
No need to get ironic---my point was merely that threaded code is hard to get right, even for a good and experienced programmer. Your first example illustrated that well, and so does your fix. Here's a comparison of runtimes:
$ time perl threads_yours.pl real 7m13.312s user 2m1.484s sys 10m8.982s $ time perl threads_mine.pl real 0m2.209s user 0m6.028s sys 0m0.604s
And here's a bit of the output:
$ uniq -c outfile | head 2275 Thread 1 1 ThreaThread 2 454 Thread 2 1 TThread 3 909 Thread 3 1 ThThread 5 454 Thread 5 1 Td 1 1364 Thread 1 1 Thread 1hread 2
This is a bog-standard Perl 5.10.1 on AMD64/i7 as it comes with Debian Squeeze¹. Either you overlooked yet another pitfall (I can't tell what it is---my point again: it looks deceptively simple but ain't) or the library implementation is buggy, either way its behavior is certainly more correct than my example's at the cost of being almost 200 times slower, but not quite correct yet.
As for the POE code, it took me indeed some 15 minutes to write:
#!/usr/bin/perl use strict; use POE qw(Component::Client::HTTP); use HTTP::Request; my @names = qw/ zezima fred bill john jack /; open my $fh, '>', 'outfile' or die $!; sub start_req { my $name = shift @names or return; POE::Kernel->post(weeble => request => response => HTTP::Request->new( GET => "http://rscript.org/lookup.php?type=track&time=6289 +9200&user=$name&skill=all" ), $name ); } POE::Session->create( inline_states => { _start => sub { POE::Component::Client::HTTP->spawn; start_req for(1 .. 5); }, response => sub { my $name = $_[ARG0]->[1]; my $result = $_[ARG1]->[0]{_content}; if($result =~ m/gain:Overall:\d+:(\d+)/isg) { print { $fh } "$name $1\n"; } elsif($result =~ m/(ERROR)/isg) { print { $fh } "$name doesn't exist \n" } else { print { $fh } "$name 0\n"; } start_req; }, }, ); POE::Kernel->run; close $fh;
I haven't used that component the last two years, so yeah, I was slow because I had to look up the defaults and how to pass the HTTP::Request object again. As long as an implementation will not trigger my pager by barfing at 4 in the morning and then look all innocent when I try and debug it, I think that's time well spent.
As for efficiency: no. The 500 do-almost-nothing threads in your code (mine didn't run long enough to register in top) need a resident set of 327 MB here (virtual size is slightly over 4 GB), I didn't try with the web scraper but I don't see how it could do any better. If I start 500 parallel requests in the POE version (well, my line here is 256 kbit on a sunny day during low tide ...), it takes 21 MB as opposed to 18 with five requests. Code-size-wise, all POE modules I have installed together are slightly over 40 kLOC including POD. I'll leave the comparison to Perl's thread code plus pthreads or whatever that builds on on your box to you. My user code is 36 non-empty lines (OK, I cheated you for two lines in the if/elsif block because that matches my style), yours is 42 so far.
¹ 5.14.2 on a newer kernel and a Phenom-II shows the same behavior, it only takes 260 MB but is even slower, I stopped it after over 20 minutes of CPU time
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^9: Consumes memory then crashs
by BrowserUk (Patriarch) on Mar 25, 2012 at 09:36 UTC | |
by mbethke (Hermit) on Mar 25, 2012 at 23:33 UTC | |
by BrowserUk (Patriarch) on Mar 26, 2012 at 03:56 UTC | |
by mbethke (Hermit) on Mar 26, 2012 at 20:52 UTC | |
by BrowserUk (Patriarch) on Mar 26, 2012 at 20:59 UTC | |
|