Re^2: LWP::Simple... Enough for Site Query & Data Download?

I've successfully downloaded and printed the content of the site I need to a file. However, the real information I need is found by going to the site and then typing in the date you want. Originally, I thought each date's data page would have a unique URL so that I could simply getprint the content from each date data page I needed. Unfortunately, these pages do not have unique URLs.

Is there any way for me to automate inputting each date I need into the text box and then hit Submit to bring up the next page?

Comment on Re^2: LWP::Simple... Enough for Site Query & Data Download? Download Code

Replies are listed 'Best First'.
Re^3: LWP::Simple... Enough for Site Query & Data Download? by Corion (Patriarch) on Jun 14, 2009 at 21:44 UTC
Maybe you want to use the full LWP::UserAgent, or the more browser-like encapsulation of it, WWW::Mechanize? I also recommend reading up on HTTP and how it works, as you'll need a bit of understanding of it if you want to automate websites.	[reply]
Re^3: LWP::Simple... Enough for Site Query & Data Download? by JavaFan (Canon) on Jun 14, 2009 at 21:54 UTC
I've successfully downloaded and printed the content of the site I need to a file. However, the real information I need is found by going to the site and then typing in the date you want. I'm confused here. In the first sentence, you claim you've had success, then the second sentence suggests you haven't had success. You can't have it both ways. Unfortunately, these pages do not have unique URLs. Have you tried doing a GET request with the CGI parameters? Many forms that are set up for POST can actually deal with GET requests as well. Is there any way for me to automate inputting each date I need into the text box and then hit Submit to bring up the next page? Well, that would be a browser issue. But not being able to do what you want with LWP::Simple doesn't mean the next step up is driving an actual browser. There are other steps as well. LWP::UserAgent, and WWW::Mechanize. They much more suitable to deal with pure HTML forms than LPW::Simple. Of course, it's also possible that the input of the text box first gets manipulated using javascript, or that one or more Ajax calls are involved. In which case, LWP::UserAgent or WWW::Mechanize still wouldn't be much improvement over LWP::Simple.	[reply]