perl: Argument list too long

sjhalani has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

we have a shell script from which we invove the perl i.e

a.sh
----
#!/bin/sh
[download]

we set a lot of environment variables here and then call perl script finally as :

$PERL_BIN/perl a.pl "$@" The perl script has the following first line i.e

#!/usr/local/bin/perl
use lib (...)
...
Usual processing ...
[download]

But when we invoke it we get the error

a.sh bin/perl: Argument list too long

The arguments we are passing are way within the limits i.e not more than 400 characters then why is the perl complaining on the argument list ? This is running on Linux box.

Is there some limit on the number of environment variables we can set ?

Comment on perl: Argument list too long Select or Download Code

Replies are listed 'Best First'.
Re: perl: Argument list too long by parv (Parson) on May 03, 2008 at 03:46 UTC
I think the error must be coming from the shell invoking perl, not perl itself. The error would not be due to setting environment variables, but what is passed to `$PERL_BIN/perl`. To confirm ... insert `set -x` just before the perl invocation line (to see the actual invocation; may want to `set +x` just after the perl line to avoid clutter in case perl call succeeds); replace `$PERL_BIN/perl` with `echo` ("argument list too long" message would still be produced); reduce the number of things stuffed in $@ (error message would enventually go away when "enough" -- consult your shell\|kernel documentation\|source -- of the arguments have been stripped away). Thanks (rest of the posters) for the clue that environment is also part of the space calculation, which I had outright dismissed in my reply.	[reply] [d/l] [select]
Re: perl: Argument list too long by pc88mxer (Vicar) on May 03, 2008 at 04:54 UTC
This is a kernel limitation. Shell arguments and environment variables are allocated from the same space and are limited to some number of pages - usually something like 32. You'll probably have to come up with another way to pass your arguments, such as reading them through STDIN or passing them in a file. For more discussion about this problem, see "Argument list too long": Beyond Arguments and Limitations Here's a script to test what the limits for `%ENV` are: Read more... (593 Bytes) On my Ubuntu box it first fails at n = 125, which corresponds to approx. 128 K (= 32 x 4K pages.)	[reply] [d/l] [select]
Re: perl: Argument list too long by arc_of_descent (Hermit) on May 03, 2008 at 07:49 UTC
This is another good link. GNU coreutils FAQ -- Rohan	[reply]
Re^2: perl: Argument list too long by quester (Vicar) on May 03, 2008 at 19:52 UTC
Thank you for that link! Oooh, sweet, unlimited environment size in Linux 2.6.23 (see the "recent news".)	[reply]
Re: perl: Argument list too long by oko1 (Deacon) on May 03, 2008 at 14:00 UTC
As many other people have pointed out, your argument list is too long. However, rather than suggesting some special arg-processing scheme (e.g., some 'xargs'-like mechanism), I would suggest that you instead reconsider your approach to the way you handle your data: if you have so many arguments on the command line that the shell is choking, then you shouldn't be presenting that data as commandline arguments. Placing it in a file, for example, and processing that file would eliminate those problems. You could also, for example, create a named pipe and have your Perl script read from it, with each argument being processed as it comes in (assuming that this is a valid way of processing your data.) In short, it's not your code that's a problem: it's your approach to handling your data. Change your approach, and the problem will go away. -- Human history becomes more and more a race between education and catastrophe. -- HG Wells	[reply]
Re: perl: Argument list too long by Helter (Chaplain) on May 03, 2008 at 04:54 UTC
I've hit this before and I believe that the "argument list" size is made up of the argument list plus all of your environment. So if you have lots of stuff set (set and setenv calls) in environment variables then this can limit what you can have on the command line...	[reply]
Re: perl: Argument list too long by jplindstrom (Monsignor) on May 03, 2008 at 13:36 UTC
Others have probably correctly identified your problem. The solution is Getopt::Long + Getopt::ArgvFile, which makes it possible to specify a command line arguments file to be used. This is a good idea in general. /J	[reply]


laziness, impatience, and hubris
	PerlMonks