Fellow monks, I have an online database that I retrieve info from all the time, and now I desperately needed to automate the task. However, I am not very familiar with the LWP module and can not get this to work. Simply put, I have a web page in which I need to initialize a SELECT, write a number to a text box, write another string to a textarea, and finally click and button and retrieve the results off the web. I know you would use something like this, but read my comments and you'll see I have lots of questions. 
$agent = LWP::UserAgent->new(); #initialize agent
my $req = POST 'http://www.database.arg', [ ]#HELP! what do I put
                                             # inside those 
                                             # brackets!
$content = $agent-request($req)->as_string; 
              #now I want to retrieve the web page the 
              #POST sends me to - and download it to 
              #save it as a file to peruse at my leisure.
              #I don't know how to do this either!

[download]
To help you answer my question, here's the HTML source of the web-page I want to automate - changed to protect the innocent. I put the comments in front of the lines for safety and paranoia. 
 
#<HTML>
#      <HEAD><TITLE>Database</TITLE></HEAD>
#
#     <HR>
#      <FORM METHOD="POST" ACTION="http:/cgi-bin/database-#result.html
+">
#      <P>
#<TABLE NOBORDER>
#<TR><TD ALIGN=LEFT>Select a subject:
#         <TD ALIGN=LEFT COLSPAN=2><SELECT #NAME=subjectspace>
#<OPTION> option 1 
#<OPTION> option 2
#<OPTION> option 3
#<OPTION SELECTED> option 4 default
#<OPTION> option 5
#</SELECT>
#             <BR><TR><TD> 
#<TR><TD ALIGN=LEFT>Number of documents:<BR>
#             <TD ALIGN=LEFT><INPUT TYPE=TEXT SIZE=5 #NAME=SubjectDocs
+>
#</TABLE>
# <P>
#      Texts to search for (separate different sentences #with a punct
+uation):<BR>
#      TEXTAREA NAME="txt1" WRAP="SOFT" ROWS=12 COLS=50>
# /TEXTAREA>
#      <P>
# INPUT TYPE="submit"  VALUE="Submit Search">
# INPUT TYPE="reset"  VALUE=" Reset to Defaults ">
#<BR>
#   </FORM>
#   <P>
#    </BODY>
#    </HTML>

[download]
Hope this helps. If you have any questions feel free to e-mail me at hhalpin@email.unc.edu. I'm really confused and will probably eventually go mad if this doesn't automate - so your help will be vastly appreciated! I also had a helluva time posting it - the only way I could get it to work was to take a '<' away from in front of TEXTAREA and the two INPUTs.

Thanks, Harry

In reply to Web Automation and POST Confusing - need help by hhalpin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.