Asynchronous HTTP requests and (digest) authentication

Ralesk has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks,

I’ve been trying to find solutions to this on the internet, but Google doesn’t seem to be favouring me today.

We’ve been using LWP for some RPC access, but since it’s blocking, we’re looking into getting a non-blocking solution to work. Two modules come to my mind, HTTP::Async and AnyEvent::HTTP. We need something that works well inside a Glib loop.

What we need is a HTTPS-capable client that can authenticate (right now we use digest auth) with the server, get the thing it needs to get, and return me the contents of the result page somehow (callback, or me having to poll it — doesn’t matter as long as it does the whole TCP/TLS/HTTP mess in the background and stops my app from hiccuping every so often.

Unfortunately none of the modules mentioned come with auth support out of the box, and it doesn’t seem to be simple at all to hack it in.

Does anyone have an idea?

Comment on Asynchronous HTTP requests and (digest) authentication

Replies are listed 'Best First'.
Re: Asynchronous HTTP requests and (digest) authentication by Corion (Patriarch) on Sep 11, 2012 at 12:07 UTC
If you're afraid of (re)implementing or copy-pasting digest auth from LWP::Authen::Digest, maybe you have luck with AnyEvent::HTTP::LWP::UserAgent or Coro::LWP or LWP::Protocol::AnyEvent::http.	[reply]
Re: Asynchronous HTTP requests and (digest) authentication by daxim (Curate) on Sep 11, 2012 at 12:07 UTC
non-blocking solution […] that works well inside a Glib loop Then the obvious module is libsoup, isn't it?	[reply]
Re^2: Asynchronous HTTP requests and (digest) authentication by Ralesk (Pilgrim) on Sep 11, 2012 at 13:59 UTC
And hahaha, no. Would be awesome, but it requires Glib::Object::Introspection which requires gobject-introspection 0.10.0 or newer, which requires a Glib slightly higher than what we unfortunately have to aim for (I’m a bit afraid of building a newer glib and toppling over the entire Gtk+ stack as a result...) Ah, the bane of having to run on older system releases :(	[reply]
Re^3: Asynchronous HTTP requests and (digest) authentication by zentara (Cardinal) on Sep 11, 2012 at 16:03 UTC
requires Glib::Object::Introspection ..... (I’m a bit afraid of building a newer glib and toppling over the entire Gtk+ stack as a result... There is no requirement for you to go to the newest libsoup gtk3 libs, except maybe some methods may be changed or missing. You could go back and get the libsoup that was compatible with your current gtk2 libs. Another option may be webkit. A Perl module exists to use it, but I don't know if it would handle your authentication requirements. You can go back into the older directories at gnome sources, to find your compatible version. It looks like libsoup goes way back into gtk2 versions before Glib::Object::Introspection even showed up on the scene. The switch from gtk2 to gtk3 occurred at GTK+ 2.24. So I would start with the libsoup with the ftp date closest to 2.24. Libs after that point are still gtk3 although they have a 2.xx number. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re^4: Asynchronous HTTP requests and (digest) authentication by Ralesk (Pilgrim) on Sep 11, 2012 at 16:30 UTC
Re^2: Asynchronous HTTP requests and (digest) authentication by Ralesk (Pilgrim) on Sep 11, 2012 at 12:22 UTC
Thanks, that looks interesting — I hope the Perl module is mature enough.	[reply]
Re: Asynchronous HTTP requests and (digest) authentication by Ralesk (Pilgrim) on Sep 21, 2012 at 15:18 UTC
Thank you all for the suggestions, here’s what I did. First of all, I wrapped things in HTTP::Request so it’s easier to manage — in particular, this also means that the `$req->uri` is a URI, which too is easier to manage. It’s also easier to construct a HEAD request to the same place this way, which is needed. Second, I tried to find where in the LWP ecosystem do we authenticate, which of course is the non-documented (and not really public) module LWP::Authen::Digest. Which of course takes (somewhat) sanitised WWW-Authenticate data (which, as it turns out from reading the source, is a somewhat unstandard header). For this, one has to use HTTP::Headers::Util qw(split_header_words) and a fine `tr///`. The rest is fairly easy, just have to emulate some structures for the `auth_header` function that don’t exist in AnyEvent::HTTP.	[reply]