in reply to A case where Mechanize works, LWP doesn't

Use something like Wireshark or Live HTTP Headers to find out what is really going on, not what you think is going on.

HTH,

planetscape
  • Comment on Re: A case where Mechanize works, LWP doesn't

Replies are listed 'Best First'.
Re^2: A case where Mechanize works, LWP doesn't
by Anonymous Monk on Feb 09, 2010 at 20:19 UTC
    Sigh. I figured I made a typo or forgot a call. Yep loaded wireshark. I guess LWP isn't as straight forward as I thought. Is LWP normally this difficult or is this just a bad one-off. I cannot imagine debugging a large program via sniffer traces.
      WWW::Mechanize uses LWP, so your question makes no sense.
        Read the original post which shows the Mechanize program works while the LWP program doesn't.
Re^2: A case where Mechanize works, LWP doesn't
by dneedles (Sexton) on Feb 10, 2010 at 16:12 UTC
    Ok. Interesting results on the Sniffer trace. It appears Mechanize is handling a redirect while LWP is not. Also the encapsulation of the POST on Mechanize is application/x-www-form-urlencoded while LWP just has variable: value in separate lines. So it appears Mechanize is doing much more work under the covers. I'll see how I can manipulate LWP to mimic Mechanize.

      Why don't you just use WWW::Mechanize? It's based on LWP, so basically to make LWP::UserAgent behave like WWW::Mechanize means just taking the code of WWW::Mechanize.

        Great questions. The reason is two fold.

        1. Limit change. The existing PERL installation doesn't use Mechanize and I would rather not introduce it primarily to keep the politics under control.

        2. POE integration. I have seen much on LWP and POE but nothing on Mechanize and POE. I'd rather go with a more proven route.

        Correct me if I am wrong (with a URL, ISDN or other tangible reference) especially on #2

        Actually found out this isn't the case. Mechanize has blocking calls. POE is more a cooperative scheduler along the lines of what Windows 3.1 was. So POE won't work well with Mechanize like it will with LWP. So I am back to using the sniffer to figure out how to code LWP with enough of Mechanize so the pages will return.