quency has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a perl script to send a request to HTTP server using LWP::UserAgent. The code is here:

-------------------------------------------------------- 
$url = 'http://151.104.234.14/voipadmin/security_chk';
$ua = new LWP::UserAgent;
$res = $ua->request(POST $url,
       Content => [ Submit => 'Submit',
                    user => 'admin',
                    passwd => 'password'
                  ]
        );

[download]
-------------------------------------------------------

When I use the microsoft network monitor to capture the packets, I found that the POST request is not sent correctly. It is sent twice. The 1st sending doesn't include the content (or data ) and the content is sent in the 2nd http request and is not sent as data. Here are the 2 frames I captured: 

--------- 1st frame POST request -----------
HTTP: POST Request (from client using port 34746)
HTTP: Request Method = POST
HTTP: Uniform ResourceIdentifier=/voipadmin/security_chk
HTTP: Protocol Version = HTTP/1.1
HTTP: Connection = Keep-Alive
HTTP: Host = 151.104.234.14
HTTP: User-Agent = libwww-perl/5.65
HTTP: Content-Length = 35
HTTP: Content-Type = application/x-www-form-urlencoded


--------- 2nd frame POST request -----------
HTTP: Submit=Submit&j_username=admin&j_password=password Request (from
+ client using p
HTTP: Request Method = Submit=Submit&user=admin&passwd=password
----------------------------------------------------

[download]

Obviously I don't get right response becasue the request is not sent correctly. The request should be sent in one packet, but look like LWP::UserAgent seperates message and send it twice. This is not correct. I check the Max_Size and it is undef (means there is no limit for the packet size). Anyone see this problem before? Any idea about this?

Thanks for your help.
Quency

20040528 Edit by Corion: Added formatting

Edit by BazB: changed all password strings to "password"

Replies are listed 'Best First'.
Re: HTTP::Request(POST, ...)
by iburrell (Chaplain) on May 28, 2004 at 23:12 UTC
    There are no requirements about how HTTP requests are put in packets. HTTP works over TCP and only deals with connection streams. Dividing into packets is up to the TCP stack of the operating system.

    Applications can influence when packets are sent by flushing buffers. LWP::UserAgent probably does two separate write for the header and the POST body, and the OS sends that as two packets.

    Either you or the network sniffer aren't giving the real HTTP header. But it looks like a proper HTTP POST, with the Content-Type, Content-Length, and url encoded body.

    Any application that wants to reliabily examine HTTP must be able to reassemble the stream from packets. Depending on clients to send the header and enough of the POST body in the same packets is unreliable.

[reply]
      Burrell, Thanks for your help. Do you know if tere is anyway to leave the HTTP packet not to be splited? How to do flush? From the packets I captured, look like the HTTP packet is not splitted correctly bcause in the 2nd packet, the header of HTTP is not correct. But I don't see any wrongly use of HTTP module. when I send the POST request, I got the error from server" HTTP Status 400 - Invalid direct reference to form login pape" look like the POST resquest is not sent correctly. Do you have any idea? Thanks a lot, Quency
[reply]

        When trying to automate any website, always capture what your browser is sending and receiving, and compare that against what you are sending and receiving.

        The error message you are giving, HTTP Status 400 - Invalid direct reference to form login pape, sounds like the server is expecting a valid Referrer: header, and you are not sending any. So, maybe try automating the page through WWW::Mechanize, which tries to behave more like a browser. Once you have that working, you can strip down the script to something only using LWP, or keep the script using WWW::Mechanize.

[reply]

Back to Seekers of Perl Wisdom