in reply to LWP::Simple... Enough for Site Query & Data Download?

My question is, does LWP::Simple, contain all the tools I need to automate visiting a site, sending a query for the data on the submitted date, searching for specific strings, and then downloading the necessary data and writing it to a file?
Well, if you mean LWP::Simple + Perl, then the answer is yes.

Does that mean life suddenly becomes easy if you use LWP::Simple, and that no other package doesn't suit solving your problem far better? That's a question that cannot be answered. You give so little information that if you had phrased your question slightly differently (for instance would LWP::Simple be a logical choice, the question wouldn't have been answerable). Now the answer is "yes, but you still may have lots to do yourself".

  • Comment on Re: LWP::Simple... Enough for Site Query & Data Download?

Replies are listed 'Best First'.
Re^2: LWP::Simple... Enough for Site Query & Data Download?
by cheech (Beadle) on Jun 14, 2009 at 21:42 UTC
    I've successfully downloaded and printed the content of the site I need to a file. However, the real information I need is found by going to the site and then typing in the date you want. Originally, I thought each date's data page would have a unique URL so that I could simply getprint the content from each date data page I needed. Unfortunately, these pages do not have unique URLs.

    Is there any way for me to automate inputting each date I need into the text box and then hit Submit to bring up the next page?

[reply]
[d/l]

      Maybe you want to use the full LWP::UserAgent, or the more browser-like encapsulation of it, WWW::Mechanize? I also recommend reading up on HTTP and how it works, as you'll need a bit of understanding of it if you want to automate websites.

[reply]
      I've successfully downloaded and printed the content of the site I need to a file. However, the real information I need is found by going to the site and then typing in the date you want.
      I'm confused here. In the first sentence, you claim you've had success, then the second sentence suggests you haven't had success. You can't have it both ways.
      Unfortunately, these pages do not have unique URLs.
      Have you tried doing a GET request with the CGI parameters? Many forms that are set up for POST can actually deal with GET requests as well.
      Is there any way for me to automate inputting each date I need into the text box and then hit Submit to bring up the next page?
      Well, that would be a browser issue. But not being able to do what you want with LWP::Simple doesn't mean the next step up is driving an actual browser. There are other steps as well. LWP::UserAgent, and WWW::Mechanize. They much more suitable to deal with pure HTML forms than LPW::Simple. Of course, it's also possible that the input of the text box first gets manipulated using javascript, or that one or more Ajax calls are involved. In which case, LWP::UserAgent or WWW::Mechanize still wouldn't be much improvement over LWP::Simple.
[reply]

In Section Seekers of Perl Wisdom