in reply to Re^2: Stateful Browsing and link extraction with AnyEvent::HTTP
in thread Async DNS with LWP

If you are currently using LWP::UserAgent, you get a HTTP::Response back. If you are using AnyEvent::HTTP, you get the data and the headers back. From the data and the headers, you can either (re)construct a HTTP::Response or do your own stuff with them directly.

If you want to handle cookies, see HTTP::Cookies which can extract cookies from a HTTP::Message.

If you want to (re)use WWW::Mechanize, see its ->update_html method and (for header handling and general message handling (the source of) its ->request method.

Replies are listed 'Best First'.
AnyEvent::DNS is effectively synchronous?
by jc (Acolyte) on Oct 05, 2010 at 11:30 UTC
    Hi, I've been trying to get parallel DNS working with AnyEvent::DNS. My first poor attempt is effectively the same as synchronous DNS because the program waits for the reply before moving on the next iteration of the loop:
    #!/usr/bin/perl -w use strict; use warnings; use AnyEvent::DNS; my ($domain); my (@domains,@condvars); my $resolver = AnyEvent::DNS::resolver; while ($domain = <>;) { # clean off newline chomp $domain; # send dns packets $resolver->resolve($domain,"*",my $condvar = AnyEvent->condvar); # receive dns packets $condvar->recv; print "$domain\n"; }
    My second attempt was slightly better in that it can send off ten different DNS packets but receives a reply from the last request ten times (not what I wanted):
    #!/usr/bin/perl -w use strict; use warnings; use AnyEvent::DNS; my ($domain); my (@domains,@condvars); my $resolver = AnyEvent::DNS::resolver; while (1) { # send dns packets for my $i (1..10) { $domain = <>; # clean off newline chomp $domain; $resolver->resolve($domain,"*",my $condvar = AnyEvent->condvar); push @condvars, $condvar; } # receive dns packets while (my $condvar = pop @condvars) { $condvar->recv; print "$domain\n"; } }
    The problem is that $condvar seems to be working like a reference to an object and so each of the 10 $condvars in the stack end up being whatever the last $condvar was. I've tried typeglobbing but this just causes compilation errors. Does anybody know how to do this right?

      You get ten times the "one domain", because you print $domain 10 times. What value would you $domain to have at which time?

      Looking at the documentation of AnyEvent::DNS, the SYNOPSIS section shows how to retrieve results from a query. I'm not sure what problems you have with that.

      If you want to do a callback-driven approach, take a look at the AnyEvent::DNS::srv subroutine. Again, the usage seems quite straightforward, as the resolved IPs get passed as parameters.

        Following is the most simple usage example from the AnyEvent::HTTP documentation. For me this program finishes without any output. Does anybody have any ideas why this doesn't work?
        #!/usr/bin/perl -w use strict; use warnings; use AnyEvent::HTTP; http_get "http://www.google.com/", sub { print $_[1] };
        Thanks, in the meantime I'd realised that myself. Just in case it's useful to anybody else I include my code so far for asynchronous resolution. As far as I can see from the documentation retrieving records from the results should be as simple as traversing an array of arrays and picking out the bits you want. e.g. A or AAAA records.
        #!/usr/bin/perl -w use strict; use warnings; use Data::Dumper; use AnyEvent::DNS; my ($domain,$START_TIME,$MAX_QUERIES,$MAX_QUEUE, $time,$done,$resolved); my (@domains,@condvars); $START_TIME = time; $MAX_QUERIES = 1000; $MAX_QUEUE = 10; $resolved = 0; # setup resolver my $resolver = AnyEvent::DNS::resolver; $resolver->max_outstanding($MAX_QUEUE); #$resolver->timeout([0,1,2]); while (1) { # send dns packets for my $i (1..$MAX_QUERIES) { $domain = <>; # clean off newline chomp $domain; $resolver->resolve($domain,"*",my $condvar = AnyEvent->condvar); push @condvars, $condvar; } # receive dns packets while (my $condvar = pop @condvars) { $resolved++ if ($condvar->recv); #warn Dumper [$condvar->recv]; } $done += $MAX_QUERIES; $time = time - $START_TIME; print "Done $done domains, $resolved resolved in $time seconds.\n"; }