Submit from browser works but LWP does not

Plankton has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I am having one hell of a time trying to get my LWP script to work with this aggravating web server using tons of javascript, jsp, and one of those awful oracle web/database solution Larry Ellison fools people into buying.

My script does this ...

    .
    .
    .
$UA->cookie_jar->add_cookie_header();
$resp = $UA->post ( $actionUrl,
        @ns_headers,
        'Content_type'    =>    'application/x-www-form-urlencoded',
        'Content' => [
            <... a bunch of stuff ...>
        ]
    );
    .
    .
    .
[download]

What that is suppose to do is create add some data to a database, but when I check the database there is nothing to be found.

When I submit the form via a browser everthing works fine, so I decided to do a tcpdump when I run my script and when I submit the form via the browser and look for differnces. And there are! When I run my script I get this tcpdump output:

POST /ditor/ditctl.do?method=save HTTP/1.1
Connection: keep-alive
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
+,text/plain;q=0.8;,image/png,*/*;q=0.5
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Host: somejunk.js.jps.com:8888
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20
+050512 Red Hat/1.7.8-1.1.3.1
Content-Length: 336
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID=1545454aae552555a48773717374620
Cookie2: $Version="1"
Keep-Alive: 300
!pu`
!pu`
content=things&blah=balhahah&morejunk=&nonsense=xyz
!pu`
HTTP/1.1 100 Continue
Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE
Date: Tue, 12 Jul 2005 18:00:00 GMT
!pu`
!pu`
HTTP/1/1 200 OK
Date: Tue, 12 Jul 2005 18:00:00 GMT
Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE
Connection: Close
Content-Type: text/html
Transfer-Encoding:chunked
<EOT>
[download]

But when I submit the form via the browser I get this output from tcpdump ...

POST /ditor/ditctl.do?method=save HTTP/1.1
Host: somejunk.js.jps.com:8888
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20
+050512 Red Hat/1.7.8-1.1.3.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
+,text/plain;q=0.8;,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Referer: http://somejunk.js.jps.com:8888/ditor/ditctl.do?stuff=stuff
Cookie: JSESSIONID=9989887adf879da987982jkd11
!pu`
!pu`
Content-Type: application/x-www-form-urlencoded
Content-Length: 384
content=things&blah=balhahah&morejunk=&nonsense=xyz
!pu`
HTTP/1.1 100 Continue
Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE
Date: Tue, 12 Jul 2005 17:29:04 GMT
!pu`
!pu`
HTTP/1/1 200 OK
Date: Tue, 12 Jul 2005 17:29:04 GMT
Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE
Content-Location: http://somejunk.js.jps.com:8888/ditor/Some.jps
Connection: Keep-Alive
Keep-Alive: tomeout=15, max=100
Content-Type: text/html
Transfer-Encoding:chunked
    ... a bunch html ...
[download]

What is going on after the form is submitted? What's this Content-Location header? How can I make my script do what the browser does??? Thanks!

Snakehandlers.net

READMORE tags added by Arunbear

Comment on Submit from browser works but LWP does not Select or Download Code

Replies are listed 'Best First'.
Re: Submit from browser works but LWP does not by ikegami (Patriarch) on Jul 12, 2005 at 20:12 UTC
Ignoring the "!pu`" and ignoring the missing blank lines in both outputs, the differences are: script: `Connection: keep-alive` brower: [nothing] This shouldn't matter. script: `Content-Length: 336` browser: `Content-Length: 384` You're not sending the same data. That could definitely cause this problem. script: `Cookie: JSESSIONID=1545454aae552555a48773717374620` browser: `Cookie: JSESSIONID=9989887adf879da987982jkd11` Differences are to be expected here, but there's no way to tell if the session is still valid. If the session is no longer valid, it could explain your problem. The difference in length of the session ids is odd, but not necessarily meaningful. script: `Cookie2: $Version="1"` brower: [nothing] This shouldn't matter. script: [nothing] browser: `Referer: http://somejunk.js.jps.com:8888/ditor/ditctl.do?stuff=stuff` This is could be important. Add this header to the script's request. There are differences in the order of the headers, but that shouldn't matter. You might have an easier time using WWW::Mechanize. It should handle just about everything for you. There are two downsides: 1) You have to install it, and 2) it won't help if the page requires JavaScript to submit.	[reply] [d/l] [select]
Re: Submit from browser works but LWP does not by BaldPenguin (Friar) on Jul 12, 2005 at 20:19 UTC
Not that I can shed too much light, but the difference in Content-length is the Content-Location header, or at least they are the same length. This is most likely a custom header for the OraWeb server to track some workspace for the java work it uses Don WHITEPAGES.COM \| INC	[reply]


Clear questions and runnable code get the best and fastest answer
	PerlMonks