The New Man has asked for the wisdom of the Perl Monks concerning the following question:

There is a form on the web, whose method is POST. I have many protein sequences to POST there, each with different sequence names. Is there a way of automating a web form to take many sequences one at a time, run each protein sequence thorugh the form, save the output in a text- format file, put that into a filehandle, then run the other sequences, one by one, always adding the resultant text output per sequence run into the 'eventual' filehandle, and so on etc. Any help will be much appreciated.
  • Comment on Automating Web Forms with a POST method

Replies are listed 'Best First'.
Re: Automating Web Forms with a POST method
by Cubes (Pilgrim) on Jul 16, 2001 at 01:56 UTC
    I think you're asking how to make a perl script pretend it's you sitting at a web browser, typing input by hand, clicking submit, and saving the resulting page somewhere.

    If that's the case, I think everything you need is in the libwww-perl bundle from CPAN. First, take a look at the HTML source for the form. Make yourself an LWP::UserAgent, put together an HTTP::Request with the URL from the form's action parameter and the appropriate content for the form's fields. CGI.pm can help you properly encode your form data. Grab the text you want from the response (HTTP::Response), and save it off to a file. Loop over a data file containing the data you want to process, appending each response to your output file.

Re: Automating Web Forms with a POST method
by Zaxo (Archbishop) on Jul 16, 2001 at 01:41 UTC

    You can use LWP::UserAgent; to mimic the POST as if it came from filling out the form. The returned page can be appended to a file with open FH,">> eventual.txt"; print FH; (assuming data in $_). Lather, rinse, and repeat.

    After Compline,
    Zaxo

    Update: Extending remarks in response to jettero. '$ man LWP::UserAgent' will get you started. 'apropos LWP' will tell you where to find more. The HTTP:Request module (basic to using LWP::UserAgent) also has a man page. To see what gets sent by the browser, modify the action in the form to read 'mailto:me@localhost' or whatever. You will see that POST consists of 'key=value' pairs, too, but in content rather than as URI modifiers.

      Just how do you get LWP::UserAgent to mimic the POST?
        try something like this (from perldoc lwpcook):
        use HTTP::Request::Common qw(POST); use LWP::UserAgent; $ua = LWP::UserAgent->new; my $req = POST 'http://www.perl.com/cgi-bin/BugGlimpse', [ search => 'www', errors => 0 ]; print $ua->request($req)->as_string;
Re: Automating Web Forms with a POST method
by synapse0 (Pilgrim) on Jul 16, 2001 at 01:31 UTC
    Yes, this is possible.
    I'm assuming you have access to make changes to the form and that you're not talking about a form on some other server that you just post to. Based on that assumption, basically, you can put at <TEXTAREA> </TEXTAREA> set on the form, so the poster can put a large amount of sequences.. when posted, have the script parse the incoming info by newlines. Process as needed and save to file.
    If you post your code-in-progress, it will be easier to help with specifics.
    -Syn0