O monks,

For my work, I end up every 6 months needing to do a particular task that requires tediously clicking through my employer's web interface to a database to retrieve 100 different data records. I'm interested in automating the process. Strangely, this web interface doesn't seem to use https anywhere, just plain http. A casual search turned up Web::Scraper, as well as a ton of other CPAN modules. Web::Scraper looks cool, but there's not much documentation. I'm also not clear on what modules would make it convenient to handle both GET and POST, as well as cookies. The canonical example that everyone seems to have in mind is ebay auctions, but that only requires going to a particular public URL and retrieving the result, without the need for any POST or cookies. For my application, I need to be able to log in with my username and password via POST, and store a cookie.

Any suggestions? Any good examples of code that does this kind of thing?

Thanks!

Ben

In reply to good CPAN modules for webscraping with GET, POST and cookies by bcrowell2

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.