CGI post truncating string?

alienhuman has asked for the wisdom of the Perl Monks concerning the following question:

Monks,

A CGI script that takes in a number of text fields and writes them to a text file seems to be truncating some of the larger strings. This script has been working fine for over a year now, and the one thing that all the truncated strings have in common is that they are exceptionally long (> 7 pages of text), so I don't think it's a problem with the script logic.

All the script itself does is grab the strings from the query params, throw them into string variables or hashes, perform regular expression type magic, and write them to a file. I don't think there is a problem with the regexp, because they are pretty harmless "remove certain characters" pattern match and substitution. And they are only truncating the fields with very long strings--I would expect a bad regexp to trash everything.

The strings seem to be getting truncated in the neighborhood of 46060 characters (~45k).

My questions:

Is there a limit to the size of a name/value pair that CGI can handle using method=POST?
Is there a limit to the buffer(?) size that perl uses for regexp?
Is there a limit to the buffer(?) size that perl uses to write to a filehandle

Thanks,
Chris

Comment on CGI post truncating string?

Replies are listed 'Best First'.
Re: CGI post truncating string? by petdance (Parson) on Nov 19, 2001 at 23:35 UTC
In order: I don't think so, but check the docs for CGI.pm. No. No. Two thoughts: Let us see some code. Maybe it IS your regex, but we can't tell. Try looking at your web server. Maybe it has some sort of limit that we can't know about without knowing what you're running. xoxo, Andy -- <megaphone> Throw down the gun and tiara and come out of the float! </megaphone>	[reply]
Re: CGI post truncating string? by cfreak (Chaplain) on Nov 20, 2001 at 01:22 UTC
Sounds like a browser issue. We were trying to do the very same thing at a company I once worked for. We found that some versions of Netscape 4.x would not allow form fields with lots of characters. I don't remember the exact number but it was some bizarre off the wall amount and if I remember correctly it wasn't always the same. I also remember that it never happened with Netscape running on FreeBSD.	[reply]
Re: CGI post truncating string? by jlongino (Parson) on Nov 20, 2001 at 00:45 UTC
If you aren't already, you might want to include: `use CGI::Carp qw(fatalsToBrowser);` [download] under your other `use CGI;` statements. It may return useful information that might otherwise go unnoticed. For an example, check out this. --Jim Update: I thought my code was error free for a year and a half.	[reply] [d/l] [select]
Re: CGI post truncating string? by Ven'Tatsu (Deacon) on Nov 20, 2001 at 00:41 UTC
Have you tried the same input in more than one browser? I don't think it's likely but you may be running into a browser specific limitation on text fields.	[reply]
Re: CGI post truncating string? by Spudnuts (Pilgrim) on Nov 20, 2001 at 00:26 UTC
Have you tested your regular expressions/processing by slurping in the values from STDIN or a file? CGI.pm is pretty handy for this. It should help you pinpoint whether the problem is interaction between the server/CGI.pm or your processing code.	[reply]
Re: CGI post truncating string? by Dr. Mu (Hermit) on Nov 20, 2001 at 01:23 UTC
This subject has come up more than once in PM. Did you remember to close the file?	[reply]
Re: CGI post truncating string? by andye (Curate) on Nov 20, 2001 at 16:00 UTC
Chris, I'm almost certain that there can be a limit on the amount of data that a web server - or at least, Apache - will accept via POST. The reason being that otherwise there's a simple denial-of-service attack possible, by just throwing loads of data at the server. (/me rummages around in the docs) If you're using Apache, the LimitRequestBody directive could be very relevant. I think CGI.pm has a similar option. (sfx: more rummaging noises) Aha! CGI.pm: Avoiding denial of service attacks You may need to set (or change) $CGI::POST_MAX. What I'd do first, if I were you, would be to check whether or not the data is actually being received in full. I.e. as soon as at arrives, dump it out to the error log. If none of the above is any help then post details of your web server, and some code if appropriate, and maybe there'll be other things to try. hth, andy.	[reply]
Re: CGI post truncating string? by tachyon (Chancellor) on Nov 20, 2001 at 19:00 UTC
I would start at the begining. Q: is the data getting to the script? Answer replace the target script with this simple dumper: `#!/usr/bin/perl -w print "Content-type: text/plain\n\n"; binmode STDIN; print while read ( STDIN, $_, 4096 );` [download] This simple script will dump the raw input your script is receiving on STDIN to the browser. It may or may not trigger a file download. Examine the output using view source or a text editor - if all your data is there start looking at your code...CGI.pm is unlikely to be the problem but first check that all the raw data is still there....continue down sequence until problem revealed. cheers tachyon s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print	[reply] [d/l]