Parsing CGI form data

Arsenal has asked for the wisdom of the Perl Monks concerning the following question:

Here's the scenario.. I need to write an automated tool for testing CGI forms. This tool needs to test all fields on the form, stuff like inputting bogus data, overflowing text input boxes, etc. I need to be able open up an arbitrary .cgi file from a remote web server, parse out the variables from the form, and generate a post to the form using a test database to assign data to the fields in the form.(actually, lots of posts) I've managed to pull the .cgi from the server, and wrote a routine to post the form data back. But, I'm having trouble parsing out the form fields. I need to obtain as much information as possible about the form such as field length, all options for drop boxes, etc. but I can't seem to write an intelligent enough parser. If someone knows of an existing program to do something like this, or could get me going in the right direction, it would be great!

Comment on Parsing CGI form data

Replies are listed 'Best First'.
(jcwren) Re: Parsing CGI form data by jcwren (Prior) on Jun 29, 2000 at 03:22 UTC
You've set a pretty daunting task for youself, here. It's pretty easy to determine what fields are on the form, using HTML::Parser, or some of the other CPAN tools. However, are you looking to parse forms where you don't know the expected content? Even a simple form, with a couple of input fields, a few radio buttons, and a couple of drop down boxes will yield thousands of possible combinations. Will the forms contains any JavaScript to perform on-the-fly input validation? If so, I'm not aware of any Perl modules that will allow you to run JavaScript in the context of a webpage. I certainly can't see a practical way to run a test for all possible combinations of an unknown webpage. Perhaps if you had a description language of some kind, and coded a template for each page to test against, you could achieve something reasonable. Testing these kinds of things dynamically are especially difficult. Your best bet may be to write a template that exercises various combinations for each known page, and know that for certain type fields you need to test putting bad e-mail addresses in e-mail fields, letters in phone number fields, etc. I know these weren't the specific answers you were looking for, but having been involved in automated testing for Windows applications (please, don't everyone start gagging), I know about some of the difficulties. --Chris	[reply]
Re: Parsing CGI form data by eduardo (Curate) on Jun 29, 2000 at 06:23 UTC
elitist punk answer follows: um... you aren't going to be able to cover the entire problem domain effectively. (meaning, you aren't going to be able to cover each and every possible answer...) could I suggest you test for what is called boundry conditions instead? write a smart-ish system to analyze the datum, and figure out what the boundry conditions are (where if's flip, where buffers get overrun...) and test those, that would be the first step i would take in attempting to make a valid effort at software testing. as a side note, i hope you understand how amazingly complex the field of software testing is, i had the amazing good fortune to study a bit under a man named roland untch, someone who taught me a great deal about software testing, just from the discussions that we had... his website storm might be of use to you, if you are interested in learing about this most amazing field... oh well :) that's that...	[reply]
Re: Parsing CGI form data by Arsenal (Novice) on Jun 29, 2000 at 04:45 UTC
Fortunately, I have discovered two good things going for me: 1 There is no javascript, just perl. 2 All the forms were auto-generated from templates. Which means that any form I check is guaranteed to have only fields that the master template it came from has. It may not have them all, but it won't have any new ones either. And, they were nice enough to include a comment just before the <FORM> tag that tells what template the form was generated from. :) Moral: Sometimes reading through product documentation pays off.	[reply]