in reply to Retrieving web pages with the LWP::UserAgent

I heartily second the advice given you by both Fletch and b10m: use WWW::Mechanize.

More specific to your stated problem: I would try using a module such as HTTP::Recorder or WWW::Mechanize::Shell to record a successful manual form submission. The output of HTTP::Recorder, for instance, can be "dropped" right into your WWW::Mechanize scripts. (leira's article "Web Testing with HTTP::Recorder" contains an excellent example of how you might want to do this.)

Another important tool for finding out what is really happening behind the scenes between server and browser is a protocol analyzer such as Ethereal.

HTH,

planetscape
  • Comment on Re: Retrieving web pages with the LWP::UserAgent

Replies are listed 'Best First'.
Re^2: Retrieving web pages with the LWP::UserAgent
by RyuMaou (Deacon) on Dec 30, 2014 at 14:39 UTC
    Thank you for pointing out HTTP::Recorder! This looks like exactly the solution to a problem I've been having getting a script to collect multiple pages out of some search results. Thank you!