sectokia has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, I can not figure out why the two programs below cause different responses from server.

The server returns a string (presenting an array) and then you can request and fetch deltas from your last request to get the array via a header.

Using LWP::UserAgent things work fine:

use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new(keep_alive => 1); my $url = "http://example.com/myUrl"; my $token = undef; while(1) { my $res = $ua->get($url, 'Start-From' => $token // "", 'X-AcceptDe +ltaUpdates' => 'true'); $token = $res->headers->{'token'} if ($res->headers->{'token'}); print $res->decoded_content if ($res->is_success); sleep 4; }
Array is: ['item1','item2'] Add ['item3'] No update Add ['item4'] No update Add ['item5'] No update ...

However using HTTP::AnyEvent I get bizarre behaviour (this is a silly way to do timing I know but I am just keeping things as simple as possible for testing):

use strict; use warnings; use AnyEvent; use AnyEvent::HTTP; my $url = "http://example.com/myUrl"; my $token = undef; while(1) { my $cv = AnyEvent::condvar; my $instance = http_request GET => $url, headers => { 'Start-From' => $token // "", 'X-AcceptDeltaUpdat +es' => 'true' }, keepalive => 1, persistent => 1, sub { my ($data, $headers) = @_; $token = $headers->{'token'} if (defined $headers->{'token +'}); print $data if ($data); $cv->send; }; $cv->recv; $cv = AnyEvent::condvar; my $timer = AnyEvent->timer(after=>4, interval => 4, cb => sub { $cv->send; }); $cv->recv; }
Array is: ['item1','item2'] No update No update No update No update No update No update Array is: ['item1','item2','item3','item4'] No update No update No update No update No update No update ...

I have tried comparing the headers sent (by setting up a HTTPD server) and all the requests seem to be identical with headers sent. So I don't understand the difference. I am basically about to resort to using wireshark to see if I can determine what is different at a TCP/HTTP level. But is there anything obvious I am missing here?

Replies are listed 'Best First'.
Re: Difference between LWP::UserAgent and AnyEvent::HTTP
by sectokia (Friar) on Sep 06, 2021 at 04:03 UTC

    After doing packet capture, I determined that AnyEvent::HTTP is sending the header 'X-AcceptDeltaUpdates' as 'X-acceptdeltaupdates', which the server ignores.

    LWP useragent retains the case you feed it.

    I believe in general the problem is that AnyEvent::HTTP should not be applying the line $hdr{lc $k} = $v to headers that start with "X-"? Anyone know HTTP spec says about this?

      The HTTP/1.0 and 1.1 RFCs both state that header names are case-insensitive. I can't find any information on X-AcceptDeltaUpdates, but IMHO the server should probably accept the lowercase version as well.

        I can't find any information on X-AcceptDeltaUpdates

        IIRC, MIME (used both for Mail and for HTTP) allows to invent non-standard "experimental" headers by prefixing them with "X-". The other end should silently ignore all unknown experimental headers.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        Yes it appears HTTP servers should do case insensitive matches to headers by specification.

        However it sort of appears by convention that HTTP libraries should not change case of headers that are specified to them.

        Lesson learned: Never trust either end!