Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm trying to use either LWP::UserAgent or the subclass WWW::Mechanize to create a program that will submit info via forms on the Lexis-Nexis search page, and save the search results. My problem *might* turn out to be one of HTML parsing -- I can't figure out what value to submit to one of the fields. In my browser window, it appears as a pull-down menu with several options to choose from: "Business & Finance", "World News", etc. But the options don't seem to even appear in the source code; instead, the options seem to consist entirely of blank space. I'm stumped -- I don't know what value to have my perl program submit! The relevant source code is below:
<select name="srccat" size="1" onchange="this.options[selectedIndex].s +elected = true;"> <OPTION VALUE="">All Available Documents&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; +&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb +sp</option> <OPTION VALUE="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp +;&nbsp</option> <OPTION VALUE="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp +;&nbsp</option> <OPTION VALUE="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp +;&nbsp</option> </select> <A HREF="" onclick="if (!submitForm(docume +nt.forms[0], 'List Sources'))return false;">Source List</A>
Can you tell what the value for the field "srccat" should be? Or can you suggest a way for me to have my program automatically select "Business & Finance" without having to know the value that corresponds to it? If you can help me solve this problem, I'll gladly donate to your site. I'm at my wits' end. Thanks a lot. Best, Julia

Replies are listed 'Best First'.
Re: Problem with automating form submission
by Anonymous Monk on Jun 16, 2004 at 19:11 UTC
    You can learn what your browser is saying (http) by using HTTP::Proxy or some other such software (like Mozilla with Live Http Headers extension), so that you can imitate it with lwp.
Re: Problem with automating form submission
by Anonymous Monk on Jun 16, 2004 at 19:15 UTC