in reply to Re^2: How to determine what the memory-limits of a CGI app are?
in thread How to determine what the memory-limits of a CGI app are? [SOLVED]

I've already suggested it twice in the other thread, but I'm going to suggest it one last time: use strace to find out what's going on under the hood.

Had you done so in the first place (i.e. compared the strace output of a working environment against the non-working case), you'd have figured out within minutes, what took you almost a week doing it your way...

An example to illustrate (I'm using DBD::Pg here, as I don't have DBD::mysql installed — but the principle is the same):

$ strace -otrace_ok perl -MDBD::Pg -e1 $ ulimit -Sv 16000 # this simulates your non-working environment $ strace -otrace_err perl -MDBD::Pg -e1 Can't load '/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/aut +o/DBD/Pg/Pg.so' for module DBD::Pg: libpq.so.3: failed to map segment + from shared object: Cannot allocate memory at /usr/local/perl/5.8.8/ +lib/5.8.8/x86_64-linux/DynaLoader.pm line 230. at -e line 0 Compilation failed in require. BEGIN failed--compilation aborted.

The relevant parts of the traces clearly show that it's the mmap which would allocate the memory for the shared lib libpq.so.3 that is failing —> "ENOMEM (Cannot allocate memory)".

Depending on the exact memory limit you set, you can make it fail sooner or later, i.e. for libpq.so.3, libnsl.so.1, libz.so.1, or whatever other shared lib dependency is involved...

----- working case ----- (...) open("/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/auto/DBD/ +Pg/Pg.so", O_RDONLY) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300_\0\0"..., +832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=93669, ...}) = 0 mmap(NULL, 1127840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, + 0) = 0x2b3b10e2a000 madvise(0x2b3b10e2a000, 1127840, MADV_SEQUENTIAL|0x1) = 0 mprotect(0x2b3b10e3b000, 1048576, PROT_NONE) = 0 mmap(0x2b3b10f3b000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE +D|MAP_DENYWRITE, 4, 0x11000) = 0x2b3b10f3b000 close(4) = 0 open("/usr/local/postgresql-7.4.5/lib64/tls/x86_64/libpq.so.3", O_RDON +LY) = -1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/tls/libpq.so.3", O_RDONLY) = - +1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/x86_64/libpq.so.3", O_RDONLY) += -1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/libpq.so.3", O_RDONLY) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0o\0\0\0"..., +832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=131049, ...}) = 0 mmap(NULL, 1158680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, + 0) = 0x2b3b10f3e000 madvise(0x2b3b10f3e000, 1158680, MADV_SEQUENTIAL|0x1) = 0 mprotect(0x2b3b10f56000, 1044480, PROT_NONE) = 0 mmap(0x2b3b11055000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE +D|MAP_DENYWRITE, 4, 0x17000) = 0x2b3b11055000 close(4) = 0 (...) ----- out of memory case ----- (...) open("/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/auto/DBD/ +Pg/Pg.so", O_RDONLY) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300_\0\0"..., +832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=93669, ...}) = 0 mmap(NULL, 1127840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, + 0) = 0x2b7780b99000 madvise(0x2b7780b99000, 1127840, MADV_SEQUENTIAL|0x1) = 0 mprotect(0x2b7780baa000, 1048576, PROT_NONE) = 0 mmap(0x2b7780caa000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE +D|MAP_DENYWRITE, 4, 0x11000) = 0x2b7780caa000 close(4) = 0 open("/usr/local/postgresql-7.4.5/lib64/tls/x86_64/libpq.so.3", O_RDON +LY) = -1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/tls/libpq.so.3", O_RDONLY) = - +1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/x86_64/libpq.so.3", O_RDONLY) += -1 ENOENT (No such file or directory) open("/usr/local/postgresql-7.4.5/lib64/libpq.so.3", O_RDONLY) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0o\0\0\0"..., +832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=131049, ...}) = 0 mmap(NULL, 1158680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, + 0) = -1 ENOMEM (Cannot allocate memory) close(4) = 0 munmap(0x2b7780b99000, 1127840) = 0 (...)

The second parameter of mmap also shows that the allocation of 1158680 bytes had been attempted.

If the system call says ENOMEM, then you can be rather sure there is not enough memory.  But you could still look at the individual mmapped sizes to see if there's something unexpectedly large among them...

Replies are listed 'Best First'.
Re^4: How to determine what the memory-limits of a CGI app are?
by locked_user sundialsvc4 (Abbot) on Mar 03, 2009 at 23:06 UTC

    Carefully repeating your process, and with the -Sv parameter of ulimit (I must have tried a different one...), I was able to produce a failure ... but in this situation the program “simply failed.” It did not produce the “error-page” output that the same program does in the CGI environment.

    The command used is:

    ulimit -Sv 32767 strace -otrace_err ./spe-cgi.pl
    and, as I said, the program “simply ends” without producing any output. It does not produce the error-page or anything else. (So I really haven't cleanly reproduced the same failure scenario... yet.)

    Nevertheless, the trace output does produce four mmap() calls in a row that each throw ENOMEM.

      But if there's a way to strace the thing in the CGI world where it now fails,...

      It's as simple as writing a shell wrapper, and have Apache call that instead of directly calling the CGI program. (Use strace's option -o, so the output doesn't end up in the error log...)

      Have you done that? What was the result?

      I really haven't cleanly reproduced the same failure scenario... yet

      Just play with different memory limits... I'm pretty sure there will be one which produces the same failure you observe in the CGI environment.

        And now the problem really becomes what to do about it. I need to discover the “wasteful” memory-allocations wherever they might be found...

        Heck, I guess I gotta start by finding memory-allocations, period ...

        What would you advise me to do here, to “trim the fat” and get this app quickly into operational condition? Truth be told, I guess I haven't had to deal with “a real memory limit” in a very long time. But it's also pretty obvious that the app is a pig.   :-D