in reply to Re^7: Consumes memory then crashs
in thread Consumes memory then crashs

Okay, so where's the code?

You just had to follow the link provided by davido to find an example. It took me 5 minutes to change it, despite I never used Mojo before: 

use 5.010;
use strict;
use warnings;

use Mojo::UserAgent;
use Mojo::IOLoop;
use Mojo::URL;

# FIFO queue
my @names = qw(zezima fred bill john jack);

# User agent following up to 5 redirects
my $ua = Mojo::UserAgent->new( inactivity_timeout => 1 );

sub url_for_name {
    my $name = shift;
    return "http://rscript.org/lookup.php?type=track&time=62899200&use
+r=$name&skill=all";
}

# Crawler
my $crawl;
$crawl = sub {
    my $id = shift;

    return unless my $name = shift @names;
    say "Looking for $name";

    # Fetch non-blocking just by adding a callback
    $ua->get(
        url_for_name($name) => sub {
            my ( $ua, $tx ) = @_;

            my $body = $tx->res->body;
            if ( $body =~ m/gain:Overall:\d+:(\d+)/i ) {
                say "$name $1";
            }
            elsif ( $body =~ m/(ERROR)/i ) {
                say "$name doesn't exist";
            }
            else {
                say "$name 0";
            }

            # Next
            $crawl->($id);
        }
    );
};

# Start a bunch of parallel crawlers sharing the same user agent
$crawl->($_) for 1 .. 4;

# Start reactor
Mojo::IOLoop->start;

[download]
And if you do, betcha it takes you 10 times longer to write; requires 10 times... less efficiently...

And that is just rubbish, especially talking about efficiency. I handled hundreds of simultaneous connections with AnyEvent::HTTP, and it didn't really consume a lot of CPU or memory, with threads it would went to swap.

Replies are listed 'Best First'.
Re^9: Consumes memory then crashs
by BrowserUk (Patriarch) on Mar 25, 2012 at 06:34 UTC

    You installed a "real time web framework" -- which looking at cpan consists of anything upto 200 modules installs 270 modules -- in order to download a few files.

    That's just dumb!

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?
[reply]

      Seems some people need an explanation for why it is dumb.

      If you prefer to use an event driven method for performing the OPs task, there is absolutely no logic in installing a 270 module "real time web framework" (nor even a 150+ module POE) to do it.

      When LWP::ParallelUA will do the job with just 9 files:

      #! perl -slw
use strict;
use LWP::Parallel::UserAgent;
use Time::HiRes qw[ time ];

sub lookup {
    my( $lookup, $resp, $prot ) = @_;
    
    my( $name ) = ${ $resp->{ _request }{ _uri } } =~ m[&user=([^&]+)&
+];

    print "Response for $name";    

    if( $lookup =~ m/gain:Overall:\d+:(\d+)/isg ) {
        print "$name $1";
    }
    elsif( $lookup =~ m/(ERROR)/isg ) {
        print "$name doesn't exist "
    }
    else{
        print "$name 0";
    }
}

my @names = do{ local @ARGV = 'firstnames.txt'; <>}; #qw(zezima fred b
+ill john jack);
chomp @names;

my $start = time;

my $pua = LWP::Parallel::UserAgent->new();
$pua->timeout( 10 );
$pua->max_req( $ARGV[ 0 ] // 10 );

$pua->register(  
    HTTP::Request->new( 'GET', 
        "http://rscript.org/lookup.php?type=track&time=62899200&user=$
+{_}&skill=all"
    ),
    \&lookup    
) for @names;

my $entries = $pua->wait;

warn time - $start;

__END__
c:\test>PUA-getnames >nul
91.8953080177307 at C:\test\PUA-getnames.pl line 44, <> line 501.

c:\test>PUA-getnames 20 >nul
82.6376140117645 at C:\test\PUA-getnames.pl line 44, <> line 501.

      [download]

      S'no quicker than threads and not that much smaller, but it does at least work!

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?
[reply]
[d/l]

      I know Chuck Moore. Chuck Moore's a great guy.

      Your Chuck Moore impression could use some work.

[reply]
        I know Chuck Moore.

        Who dat!

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?
[reply]

In Section Seekers of Perl Wisdom