Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Redirect after POST behavior in LWP::UserAgent differs from Netscape's?

by gregorovius (Friar)
on Mar 16, 2001 at 01:01 UTC ( [id://64784] : perlquestion . print w/replies, xml ) Need Help??

gregorovius has asked for the wisdom of the Perl Monks concerning the following question:

Dear fellow Monks,

I'm using LWP::UserAgent to simulate users on a website. The problem I have is that UserAgent behavior after POSTing appears to be different to Netscape's.

When my web application instructs Netscape to redirect (by sending back a 302 Moved message) Netscape will generate a new request which will be a GET request, and discarding the POST parameters of the original request. Netscape, thus, transforms the POST request into a GET request with no content after redirection.

LWP::UserAgent, to begin with, won't follow redirects after posting unless you override the LWP::UserAgent::redirects_ok method to return True for POST requests. Once you do this its behavior for POST redirects will be to issue a new POST request with the same payload (content) of the original request. UserAgent's behavior, thus, is to reissue the exact same request, to the new URL.

I don't know what the HTTP RFC says, but I guess it wouldn't matter, since Netscape's has to be the de-facto standard.

This difference in behaviors is preventing me from reproducing exact user request sequences because my web application assumes that redirects after posting carry no parameters, which is not the case for UserAgent redirects.

I've modified LWP::UserAgent to mimick Netscape's behavior, and now my test sequences pass.

My questions are, do you think this is a bug in LWP::UserAgent, and what the RFC says? A coworker of mine suggests that I submit a patch to LWP for it to do "Netscape emulation". Who do you think is right? I guess the behavior of IE is the same as Netscape's since our web application works fine on it, too.



Replies are listed 'Best First'.
Re: Redirect after POST behavior in LWP::UserAgent differs from Netscape's?
by extremely (Priest) on Mar 16, 2001 at 01:24 UTC
    don't know what the HTTP RFC says, but I guess it wouldn't matter, since Netscape's has to be the de-facto standard.

    The RFC says that allowing a redirect to a POST is unsafe and that clients shouldn't do it. To do so is evil and guaranteed to act weird with all browsers. Some will convert to GET style and happily send your data to the error page you got because the CGI is broken. Others ask and some fail silently.

    I think that you will find people rather unhappy with you offering to break compliance with a standard. Make sure the patch you offer is both optional and defaults to off.

    Sarcasm about Netscape being "standard" removed before posting =) =)

    $you = new YOU;
    honk() if $you->love(perl)

      Sarcasm about Netscape being "standard" removed before posting =) =)
      I've often said (and gotten away with) in my columns the phrase "non-standard browsers, such as Netscape and IE".

      -- Randal L. Schwartz, Perl hacker

      But people use those undocumented quirks in browser implementation. I don't think we could ever banish reliance on such non-standard features, so maybe the best way to deal with them would be to incorporate them to the RFC in substitution of the original spec (I'm not talking specifically about this one quirk, which you say is unsafe; BTW, would you dwell into why it is so?).

      PerlMonks itself, doesn't it rely on Netscape's redirect after POST behavior?

      Also, consider that Perl is a good example of the implementation being also the standard, a system that has worked pretty well, IMHO.

        Well, the main "danger" is that you could be directed to a site other than the one you felt safe giving your data too. If you hit a post to Amazon's order form and the last page redirected you to Barnes and Noble's page for the purchase. Worse, if the error document of the site was "tweaked" and the CGI removed you might wind up pouring your data into someone else's logs.

        Also, please don't confuse redirect AFTER post with redirecting THE post. You shouldn't be conned into sending data to someone you didn't expect. Where the browser goes after the data is received properly is something else entirely.

        Read for yourself: RFC 2616 Sec 10.3.2 and and you might look at most of Sec 15 as well.

        $you = new YOU;
        honk() if $you->love(perl)

      Newer versions of Netscape will warn you about being redirected with POST before allowing it to happen; I believe that the same goes for recent MSIE's.