joinersw has asked for the wisdom of the Perl Monks concerning the following question:

I'm hoping someone has run into this previously...

os=WinXP
perl=active perl 5.6.1

we have quite a few environment varibles defined for
our application...under XP, using the cmd.exe shell,
the limits for a single variables length is somewhere near
an 8K limit, while the entire environment space is
65MB...this comes from the microsoft knowledgebase.

Using 'wc' (word count - gnu util), the total env space
that I am occupying contains roughly 30,000 charaters.
If I extend this by adding a new variable in a perl
script using $ENV, and I exceed 31,060 or there about, I
experience a failure to execute the system call
all together...however all other perl execution is fine...
I can print the variables that I have just set, etc. The
system call would be something simple like system("dir");

I have tried backticks and using exec. No luck.

I have examined PERL5SHELL and it points to cmd.exe which
should support much larger env sizes accroding to
microsoft.

Any ideas as to why I am running up against this limit?

Replies are listed 'Best First'.
Re: environment limitations
by shenme (Priest) on Oct 01, 2004 at 22:39 UTC
    Look at KB article 830473, where they mention a number close to what you mention:
    Even though the Win32 limitation for environment variables is 32,767 characters, ...
    While they seem in some places to mean this as a limit for _one_ variable, it does make me suspicious that somewhere this is a limit for the combination of variables. For instance, in one article for Win2K they explicitly say "The default environment size is 256 bytes, with a maximum possible size of 32 KB."

    But I have to say I am ... amazed ... at your IMO excessive use of environment variables to pass values to your programs. I can only think of two explanations: a good idea gone bad through mindless repetition, or you must share this data with other applications that can _only_ use environment variables. If not the latter case then consider re-thinking your use of environment variables. As you can now see, you have ridden this beast into the ground.

      Thanks for the response. And I can appreciate the curiosity as to why the use of variables

      1. the use of variables is due to the application running on both unix and windows platforms and using the same source to configure the application environment. It works quite well and centralizes the maintenance of a large number of clients in a hybrid os, global environment.

      2. the information that you've supplied I've also seen...however, I have also seen on the microsoft site the following...

    • The maximum individual environment variable size is 8192bytes.
    • The maximum total environment variable size for all variables, which includes variable names and the equal sign, is 65,536KB.

      http://www.microsoft.com/windowsxp/home/using/productdoc/en/default.asp?url=/windowsxp/home/using/productdoc/en/ntcmds_shelloverview.asp

      This is the source of my confusion
        Well, I think I went overboard in wanting to make up for being less than helpful. I wanted to at least confirm your symptoms, with the idea of lending more credence to the apparent fact that Microsoft lies (gasp!) about limits vis-a-vis environment variables.

        So I typed in a simple Perl program as you indicated:

        #!/usr/bin/perl -w # vim:ft=perl:ts=4:sw=4:et:is:hls:ss=10: use strict; use warnings; my $the_cmd = 'dir'; printf "About to call system('%s') ...\n", $the_cmd; my $rc = system( $the_cmd ); printf "Return code from system() call was %d (0x%04X)\n", $rc, $r +c;
        Then (and in part to remind myself why Perl is *so* much better) I typed in a batch file (WinXPSP2) to test with:

        (oh faint of heart - shield your eyes!)

        @echo off :menu1 echo c)lear all echo f)ill a bunch echo l)ist them echo s)ize them echo r)un Perl program echo q)uit set /p zans="what to do? [cflqrs] " if "%zans%X"=="X" goto menu1 if /i "%zans%"=="c" goto docl1 if /i "%zans%"=="f" goto dofl1 if /i "%zans%"=="l" goto doli1 if /i "%zans%"=="q" goto doqu1 if /i "%zans%"=="r" goto doru1 if /i "%zans%"=="s" goto dosz1 goto menu1 :docl1 set zcount=350 set zcounter=1 :clagain rem echo I see %zcounter% and %zcount% if %zcount% LSS 1 goto :menu1 if not defined zenv%zcounter% goto menu1 rem echo Once again, doing zenv%zcounter% ... set zenv%zcounter%= set /a zcounter+=1 set /a zcount-=1 goto clagain :dofl1 --------------------------------------------------- set /p zans="how many should I fill? " if "%zans%X"=="X" goto menu1 set zcount=%zans% echo Will fill %zcount% environment variables with 100-char values :flagain if %zcount% LSS 1 goto :menu1 set zenv%zcount%=a123456789b123456789c123456789d123456789e123456789f12 +3456789g123456789h123456789i123456789j123456789 set /a zcount-=1 goto flagain :doli1 set zenv goto menu1 :dosz1 set zenv >zenv.size set >all.env.size dir zenv.size all.env.size goto menu1 :doru1 perl -w joinersw1a.pl goto menu1 :doqu1 echo okay, quitting, bye! goto :eof rem vim:ft=dosbatch:ts=2:sw=2:et:is:hls:ss=10:
        I then ran it, asking for increasing numbers of environment variables to be filled with short 100-char strings. I did hit a limit fairly quickly. Below are snippets from the output:
        how many should I fill? 260
        Will fill 260 environment variables with 100-char values
        10/03/2004  23:33            29,983 all.env.size
        10/03/2004  23:33            28,492 zenv.size
        About to call system('dir') ...
        CGI-Application-MailPage-1.2.tar.gz  hexdmpr.pl~      murugu1.pl
            ::      ::     ::
        hexdmpr.pl~                          murugu1.pl
        Return code from system() call was 0 (0x0000)
        
        how many should I fill? 265
        Will fill 265 environment variables with 100-char values
        10/03/2004  23:34            30,533 all.env.size
        10/03/2004  23:34            29,042 zenv.size
        About to call system('dir') ...
        CGI-Application-MailPage-1.2.tar.gz  hexdmpr.pl~      murugu1.pl
            ::      ::     ::
        hexdmpr.pl~                          murugu1.pl
        Return code from system() call was 0 (0x0000)
        
        how many should I fill? 269
        Will fill 269 environment variables with 100-char values
        10/03/2004  23:36            30,973 all.env.size
        10/03/2004  23:36            29,482 zenv.size
        About to call system('dir') ...
        CGI-Application-MailPage-1.2.tar.gz  hexdmpr.pl~      murugu1.pl
            ::      ::     ::
        hexdmpr.pl~                          murugu1.pl
        Return code from system() call was 0 (0x0000)
        
        how many should I fill? 270
        Will fill 270 environment variables with 100-char values
        10/03/2004  23:35            31,083 all.env.size
        10/03/2004  23:35            29,592 zenv.size
        About to call system('dir') ...
        Can't spawn "cmd.exe": No such file or directory at joinersw1a.pl line 9.
        Return code from system() call was 65280 (0xFF00)
        
        All I can really offer is that I can reproduce the problem, and quite close to your problem specification. I can't find any documentation at Microsoft that relates, other than their statements about 65MB max limits. (lies!)

        But I do feel you should investigate one or more of the several configuration file solutions. I just have to believe it will be more manageable now and in the future.
        (once over that little hiccup of totally changing the way you do things (ouch!))

Re: environment limitations
by dba (Monk) on Oct 02, 2004 at 17:05 UTC
    Probably you must have already thought about it... Curious as to why a app.ini or app.config file won't do the work ?
      Use your environment space to write the path of your new ini file. That should work multiplatform. Don't get into M$ knowledge base labrinth. It's too obscure, in my point of view.

      .{\('v')/}
      _`(___)' __________________________
      Wherever I lay my KNOPPIX disk, a new FREE LINUX nation could be established.
Re: environment limitations
by lithron (Chaplain) on Oct 02, 2004 at 20:04 UTC
    could you compress and base64 the strings to shorten them some? Then have the applications that want to read the data undo the base64 and the compression?