in reply to Re: How to determine what the memory-limits of a CGI app are?
in thread How to determine what the memory-limits of a CGI app are? [SOLVED]

As I have related in several other painful current threads, I know that the final failure-message is: 

libz.so.1: failed to map segment from shared object: 
Cannot allocate memory

[download]

(This is the message I wrangled from the bootstrap call in DBD/mysql.pm.)

Furthermore, I know that if I put libz into a static-linked driver... another library-name is promptly listed in the same message. So, “it's really nothing to do with the library.”

I have conclusively determined that the libraries are being requested from the proper locations.

I also know that the program works correctly:

Furthermore, the program goes-down gracefully. It displays its own error-response page and actually ends normally:   Perl does not complain of running out of memory! Yes, it correctly finds mysql.so, at the correct place and none other, but it will not load it.

Replies are listed 'Best First'.
Re^3: How to determine what the memory-limits of a CGI app are?
by almut (Canon) on Mar 03, 2009 at 22:49 UTC

    I've already suggested it twice in the other thread, but I'm going to suggest it one last time: use strace to find out what's going on under the hood.

    Had you done so in the first place (i.e. compared the strace output of a working environment against the non-working case), you'd have figured out within minutes, what took you almost a week doing it your way...

    An example to illustrate (I'm using DBD::Pg here, as I don't have DBD::mysql installed — but the principle is the same):

    $ strace -otrace_ok  perl -MDBD::Pg -e1 
$ ulimit -Sv 16000    # this simulates your non-working environment
$ strace -otrace_err perl -MDBD::Pg -e1 
Can't load '/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/aut
+o/DBD/Pg/Pg.so' for module DBD::Pg: libpq.so.3: failed to map segment
+ from shared object: Cannot allocate memory at /usr/local/perl/5.8.8/
+lib/5.8.8/x86_64-linux/DynaLoader.pm line 230.
 at -e line 0
Compilation failed in require.
BEGIN failed--compilation aborted.

    [download]

    The relevant parts of the traces clearly show that it's the mmap which would allocate the memory for the shared lib libpq.so.3 that is failing —> "ENOMEM (Cannot allocate memory)".

    Depending on the exact memory limit you set, you can make it fail sooner or later, i.e. for libpq.so.3, libnsl.so.1, libz.so.1, or whatever other shared lib dependency is involved... 

    ----- working case -----
(...)
open("/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/auto/DBD/
+Pg/Pg.so", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300_\0\0"..., 
+832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=93669, ...}) = 0
mmap(NULL, 1127840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4,
+ 0) = 0x2b3b10e2a000
madvise(0x2b3b10e2a000, 1127840, MADV_SEQUENTIAL|0x1) = 0
mprotect(0x2b3b10e3b000, 1048576, PROT_NONE) = 0
mmap(0x2b3b10f3b000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE
+D|MAP_DENYWRITE, 4, 0x11000) = 0x2b3b10f3b000
close(4)                                = 0
open("/usr/local/postgresql-7.4.5/lib64/tls/x86_64/libpq.so.3", O_RDON
+LY) = -1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/tls/libpq.so.3", O_RDONLY) = -
+1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/x86_64/libpq.so.3", O_RDONLY) 
+= -1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/libpq.so.3", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0o\0\0\0"..., 
+832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=131049, ...}) = 0
mmap(NULL, 1158680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4,
+ 0) = 0x2b3b10f3e000
madvise(0x2b3b10f3e000, 1158680, MADV_SEQUENTIAL|0x1) = 0
mprotect(0x2b3b10f56000, 1044480, PROT_NONE) = 0
mmap(0x2b3b11055000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE
+D|MAP_DENYWRITE, 4, 0x17000) = 0x2b3b11055000
close(4)                                = 0
(...)

----- out of memory case -----
(...)
open("/usr/local/perl/5.8.8/lib/site_perl/5.8.8/x86_64-linux/auto/DBD/
+Pg/Pg.so", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300_\0\0"..., 
+832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=93669, ...}) = 0
mmap(NULL, 1127840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4,
+ 0) = 0x2b7780b99000
madvise(0x2b7780b99000, 1127840, MADV_SEQUENTIAL|0x1) = 0
mprotect(0x2b7780baa000, 1048576, PROT_NONE) = 0
mmap(0x2b7780caa000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXE
+D|MAP_DENYWRITE, 4, 0x11000) = 0x2b7780caa000
close(4)                                = 0
open("/usr/local/postgresql-7.4.5/lib64/tls/x86_64/libpq.so.3", O_RDON
+LY) = -1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/tls/libpq.so.3", O_RDONLY) = -
+1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/x86_64/libpq.so.3", O_RDONLY) 
+= -1 ENOENT (No such file or directory)
open("/usr/local/postgresql-7.4.5/lib64/libpq.so.3", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0o\0\0\0"..., 
+832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=131049, ...}) = 0
mmap(NULL, 1158680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4,
+ 0) = -1 ENOMEM (Cannot allocate memory)
close(4)                                = 0
munmap(0x2b7780b99000, 1127840)         = 0
(...)

    [download]

    The second parameter of mmap also shows that the allocation of 1158680 bytes had been attempted.

    If the system call says ENOMEM, then you can be rather sure there is not enough memory.  But you could still look at the individual mmapped sizes to see if there's something unexpectedly large among them...

[reply]
[d/l]
[select]

      Carefully repeating your process, and with the -Sv parameter of ulimit (I must have tried a different one...), I was able to produce a failure ... but in this situation the program “simply failed.” It did not produce the “error-page” output that the same program does in the CGI environment.

      The command used is:

      ulimit -Sv 32767
strace -otrace_err ./spe-cgi.pl

      [download]
      and, as I said, the program “simply ends” without producing any output. It does not produce the error-page or anything else. (So I really haven't cleanly reproduced the same failure scenario... yet.)

      Nevertheless, the trace output does produce four mmap() calls in a row that each throw ENOMEM.

        I really haven't cleanly reproduced the same failure scenario... yet

        Just play with different memory limits... I'm pretty sure there will be one which produces the same failure you observe in the CGI environment.

[reply]
        But if there's a way to strace the thing in the CGI world where it now fails,...

        It's as simple as writing a shell wrapper, and have Apache call that instead of directly calling the CGI program. (Use strace's option -o, so the output doesn't end up in the error log...)

        Have you done that? What was the result?

[reply]
[d/l]
Re^3: How to determine what the memory-limits of a CGI app are?
by locked_user sundialsvc4 (Abbot) on Mar 03, 2009 at 21:29 UTC

    Taking the ulimit suggestion above, I get the following output: 

    core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 36864
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 38912
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) 36864
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 10
max user processes              (-u) 16
virtual memory          (kbytes, -v) 36864
file locks                      (-x) unlimited

    [download]

    By my reading, then, the program has 36 megabytes of available RAM, and an 8 megabyte stack. Actually more than the hosting-service promised to deliver, and unquestionably more than enough resources to run this program.

    And yet... it doesn't run. If it claims to be “out of memory,” all I can say is ... something having to do with cow excrement.

    I still don't know the real reason why it cannot allocate memory, but I'm certain that it's not actually “running out of” anything. Something is different between this case and all others... I just can't determine what it is.

In Section Seekers of Perl Wisdom