Input Filtering Requirements

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm setting up a few scripts to grab user input and store it in CSV files. The user input will consist of personal information such as name and email address and other information from survey forms consisting mostly of radio buttons and checkboxes.

What I'm trying to ensure is that all incoming data is filtered properly. Since I'm just storing the data, my primary concern is the length of the data. How is the best way to filter this? Just use Perl's length function as in if (length($blah) > $max_length) { error(); }?

I'm also validating some data using Email::Valid and it's nice little regex. Is there any other validation/checking I should be doing? I'm so used to filtering out questionable characters, but is there any point if I just storing this in a file? I'm not doing anything silly like using the input to name the file, so filtering combinations such as "../" doesn't appear necessary.

Thank you in advance for your advice.

Comment on Input Filtering Requirements Download Code

Replies are listed 'Best First'.
Re: Input Filtering Requirements by AcidHawk (Vicar) on Aug 10, 2003 at 11:51 UTC
Have you considered numeric values in the name fields.. Re: form validation has a neat little routine to look after this.. There is also How do I detect and handle empty form fields? which might be of use. If it is just length you are looking for then Re: Length validation spells it out nicely.. Some other links that might prove useful Data::FormValidator validating string length with a regular expression Validating incoming CGI form data more links - CGI parameter Validation ----- Of all the things I've lost in my life, its my mind I miss the most.	[reply]
Re: Input Filtering Requirements by blue_cowdawg (Monsignor) on Aug 10, 2003 at 14:15 UTC
Is there any other validation/checking I should be doing? The short answer is: "it depends." What are your program requirements? Who is the customer and who is the expected user base? As evil as I consider JavaScript, when I am doing CGI programming and have an input form that is going to be filled out by the general public I have a JavaScript validator check EVERYTHING on the form for validity. If the input field is a mandatory field then I make sure it is not only valid but I force the user to fill it out properly if it is wrong. If it is not a mandatory field then I blank it out if it is wrong and move on. By using JavaScript I'm not "bothering" the hosting system with garbage and keeping data validation out on the user's browser. There is a second level of data validation that occurs that is best kept on the hosting machine. This has to do with business logic and for a lack of better phrase account information validation. For instance for an E-Commerce type application I know that Joe Blow userid jblow123 does not have an account number with me 897654 so if he tries to claim that account is his I can rebuff him. So, in my opinion the simple question you pose is not as simple as it seems and my short answer of "it depends" is a valid one. Summing my answer up I would say that you need to go back to the customer (or you boss or whoever you are writing this for) and negotiate exactly what the requirements are and from that determine what data is valid and what is not and what you even care to validate. Peter @ Berghold . Net Sieze the cow! Bite the day! Nobody expects the Perl inquisition! Test the code? We don't need to test no stinkin' code! All code posted here is as is where is unless otherwise stated. Brewer of Belgian style Ales	[reply]
Re: Re: Input Filtering Requirements by Anonymous Monk on Aug 10, 2003 at 16:31 UTC
By using JavaScript I'm not "bothering" the hosting system with garbage and keeping data validation out on the user's browser. I sure hope you're validating that data again on the server side. Depending on the client to validate the data it's sending is a recipe for disaster.	[reply]
Re: Re: Re: Input Filtering Requirements by blue_cowdawg (Monsignor) on Aug 10, 2003 at 21:33 UTC
I sure hope you're validating that data again on the server side. Don't worry about that... I am sufficiently paranoid about my world to check things as many times as I have to in order to be comfortable. If there is a will there is a way... Besides.. I'm sure you have heard the saying: "As soon as you make something idiot proof they go and devlop the next revision of idiot." Peter @ Berghold . Net Sieze the cow! Bite the day! Nobody expects the Perl inquisition! Test the code? We don't need to test no stinkin' code! All code posted here is as is where is unless otherwise stated. Brewer of Belgian style Ales	[reply]
Re: Input Filtering Requirements by waswas-fng (Curate) on Aug 11, 2003 at 04:15 UTC
I'm so used to filtering out questionable characters, but is there any point if I just storing this in a file? I'm not doing anything silly like using the input to name the file, so filtering combinations such as "../" doesn't appear necessary. This all depends on what you are doing. If this is an interactive script then by all means verify input and request fixes where needed. If not interactive and this data in the CSV is going to be used by another program that does not expect tainted data by all means verify the data and reject tainted entries. -Waswas	[reply]