in reply to Stateful Browsing and link extraction with AnyEvent::HTTP
in thread Async DNS with LWP

I had thought about using AnyEvent::DNS but there don't seem to be any obvious ways of getting LWP to use its results rather than doing its own synchronous resolution (via the OS).

Surely, if you resolve the domain name yourself (asynchronously or not), and then supply the resolved dotted decimal as part of the url you supply to LWP, it won't have to, or be able to, do the resolution again itself?

(I know; you don't like being called Shirley:)

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy
  • Comment on Re: Stateful Browsing and link extraction with AnyEvent::HTTP

Replies are listed 'Best First'.
Re^2: Stateful Browsing and link extraction with AnyEvent::HTTP
by Corion (Patriarch) on Oct 05, 2010 at 09:24 UTC

    I think it's not that simple because LWP needs to generate the appropriate Host: header. But if you generate that header in your code, I think you should be able to reuse most of what LWP already supplies.

    Alternatively, maybe you can even supply the appropriate opened (blocking) socket to LWP after having made an (asynchronous) connection to the host. But that involves source diving, surely.

[reply]
[d/l]
Re^2: Stateful Browsing and link extraction with AnyEvent::HTTP
by Anonymous Monk on Oct 05, 2010 at 10:13 UTC
    It will work for non-authenticated requests easy, but not for authenticated ones
    Read more... (3 kB)
[reply]
[d/l]

      This seems to work ok? (Note the extra line!): 

      #!/usr/bin/perl --

use strict;
use warnings;

use WWW::Mechanize 1.54;

my $ua = WWW::Mechanize->new;
$ua->add_handler(
    "request_send",  sub {
        $_[0]->content('');
        $_[0]->dump;
        return
    }
);

$ua->add_handler(
    "response_done", sub {
        $_[0]->content('');
        $_[0]->dump;
        return
    }
);

my $uri = URI->new('http://jigsaw.w3.org/HTTP/Basic/');
$ua->credentials( $uri->host_port, qw' test guest guest ');

$ua->get( $uri );

print "\n--- SPECIFY HOST MANUALLY ---\n";

my $host = $uri->host( $ua->res->header('Client-peer') );
$ua->credentials( $uri->host_port, qw' test guest guest '); ### ADDED 
+##
$ua->get( $uri, Host => $host );

__END__
GET http://jigsaw.w3.org/HTTP/Basic/
Accept-Encoding: gzip
User-Agent: WWW-Mechanize/1.66

      [download]
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        Why yes it does, pure genius :D
[reply]
Re^2: Stateful Browsing and link extraction with AnyEvent::HTTP
by james2vegas (Chaplain) on Oct 05, 2010 at 09:26 UTC
    The web server may have multiple named virtual servers on the same IP, in which case it will check the Host passed in with the request to determine what webroot to use
[reply]

      Just supply the Host: header yourself, you have the appropriate information.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom