memory usage on x86

powerman has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: memory usage on x86_64 by Eliya (Vicar) on Feb 15, 2012 at 22:34 UTC
"softlimit `-m`" also limits the maximum virtual address space (i.e. it calls setrlimit(2) on the resource RLIMIT_AS¹⁾ — see the man page for details). So what you're testing here is the virtual memory usage. The virtual memory usage of a program, however, is not necessarily a useful measure of how much memory a program actually currently "needs". It's more a measure of the address space that the program can potentially address (without allocating more), not of how much physical memory it actually has in use — the resident set size (RSS) is a better measure of the latter. In other words, I wouldn't be worried too much :) For comparison, my 64-bit Perl 5.12.4 on Ubuntu uses: `$ perl -le 'use version; print $version::VERSION; system "ps -p $$ -o +rss,vsz,comm"' 0.82 RSS VSZ COMMAND 2476 24236 perl` [download] ___ ¹⁾ `$ strace -esetrlimit softlimit -m 25000000 perl -e1 setrlimit(RLIMIT_DATA, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) = +0 setrlimit(RLIMIT_STACK, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) = + 0 setrlimit(RLIMIT_MEMLOCK, {rlim_cur=641024, rlim_max=641024}) = 0 setrlimit(RLIMIT_AS, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) = 0` [download]	[reply] [d/l] [select]
Re^2: memory usage on x86_64 by locked_user sundialsvc4 (Abbot) on Feb 15, 2012 at 23:56 UTC
IMnsHO, this is a crucial observation. The virtual memory manager is always “lazy.” If the memory is available, and if there is no “pressure” actually being exerted upon the virtual-memory resource, then the VMM will make absolutely no effort to clean up. After all, it is perfectly plausible that, someday, you might ask for that same bit of information again. It is to the system’s advantage to presume that you will do so. VMM’s are designed to respond only to actual memory pressure. Like a typical bachelor, it won’t load the dishwasher until three minutes before its girlfriend is due to knock on the door. And, on our typically fast and typically capacious “developer” machines, that rarely happens. (Memory pressure, I mean ... not girlfriends. We hope.) Consequently, memory requirements are customarily reported too large.
Re^2: memory usage on x86_64 by powerman (Friar) on Feb 16, 2012 at 07:56 UTC
You right about RSS/VSZ: `### x86 # perl -le 'use version; print $version::VERSION; system "ps -p $$ -o +rss,vsz,comm"' 0.96 RSS VSZ COMMAND 2068 5552 perl ### x86_64 # perl -le 'use version; print $version::VERSION; system "ps -p $$ -o +rss,vsz,comm"' 0.96 RSS VSZ COMMAND 2468 20044 perl` [download] But thing is, virtual or resident, memory limits should be significantly increased on 64-bit systems. Real issue I had because of this was related to qmail (which by default run under setlimit) filter - perl script which implement greylisting. As you see, when perl doesn't get enough memory it's error messages ain't very informative. So, that I got in system logs is useless messages about my locale, coming from unknown source - and at that time my system rejects incoming emails because part of qmail execution chain was crashed. And it's still doesn't clear for me, is so much virtual memory requested by perl on 64-bit system is a bug in perl, or correct behaviour.	[reply] [d/l]
Re^3: memory usage on x86_64 by Eliya (Vicar) on Feb 16, 2012 at 10:43 UTC
And it's still doesn't clear for me, is so much virtual memory requested by perl on 64-bit system is a bug in perl, or correct behaviour. If you have doubts things are as they should be, you can investigate the virtual memory mappings of the process via `/proc/<PID>/maps` (or `/proc/self/maps` from within the process). See also `proc(5)` for a short explanation of the columns. The following snippet might help with computing sizes: `#!/usr/bin/perl open my $maps, "<", "/proc/self/maps" or die $!; my $sum; while (<$maps>) { /^(\S+)-(\S+)/; my $size = hex($2)-hex($1); $sum += $size; printf "%8d : %s", $size, $_; } printf "total: %dk\n", $sum/1024;` [download] On my current system (64-bit openSUSE 11.1, Perl 5.12.3), it shows 1454080 : 00400000-00563000 r-xp 00000000 08:03 1209182 + /usr/local/perl/5.12.3/bin/perl 4096 : 00762000-00763000 r--p 00162000 08:03 1209182 + /usr/local/perl/5.12.3/bin/perl 16384 : 00763000-00767000 rw-p 00163000 08:03 1209182 + /usr/local/perl/5.12.3/bin/perl 270336 : 00767000-007a9000 rw-p 00767000 00:00 0 + [heap] 1372160 : 7fcc9341c000-7fcc9356b000 r-xp 00000000 08:03 960813 + /lib64/libc-2.9.so 2097152 : 7fcc9356b000-7fcc9376b000 ---p 0014f000 08:03 960813 + /lib64/libc-2.9.so 16384 : 7fcc9376b000-7fcc9376f000 r--p 0014f000 08:03 960813 + /lib64/libc-2.9.so 4096 : 7fcc9376f000-7fcc93770000 rw-p 00153000 08:03 960813 + /lib64/libc-2.9.so 20480 : 7fcc93770000-7fcc93775000 rw-p 7fcc93770000 00:00 0 90112 : 7fcc93775000-7fcc9378b000 r-xp 00000000 08:03 960839 + /lib64/libpthread-2.9.so 2097152 : 7fcc9378b000-7fcc9398b000 ---p 00016000 08:03 960839 + /lib64/libpthread-2.9.so 4096 : 7fcc9398b000-7fcc9398c000 r--p 00016000 08:03 960839 + /lib64/libpthread-2.9.so 4096 : 7fcc9398c000-7fcc9398d000 rw-p 00017000 08:03 960839 + /lib64/libpthread-2.9.so 16384 : 7fcc9398d000-7fcc93991000 rw-p 7fcc9398d000 00:00 0 8192 : 7fcc93991000-7fcc93993000 r-xp 00000000 08:03 960847 + /lib64/libutil-2.9.so 2093056 : 7fcc93993000-7fcc93b92000 ---p 00002000 08:03 960847 + /lib64/libutil-2.9.so 4096 : 7fcc93b92000-7fcc93b93000 r--p 00001000 08:03 960847 + /lib64/libutil-2.9.so 4096 : 7fcc93b93000-7fcc93b94000 rw-p 00002000 08:03 960847 + /lib64/libutil-2.9.so 53248 : 7fcc93b94000-7fcc93ba1000 r-xp 00000000 08:03 960817 + /lib64/libcrypt-2.9.so 2097152 : 7fcc93ba1000-7fcc93da1000 ---p 0000d000 08:03 960817 + /lib64/libcrypt-2.9.so 4096 : 7fcc93da1000-7fcc93da2000 r--p 0000d000 08:03 960817 + /lib64/libcrypt-2.9.so 4096 : 7fcc93da2000-7fcc93da3000 rw-p 0000e000 08:03 960817 + /lib64/libcrypt-2.9.so 188416 : 7fcc93da3000-7fcc93dd1000 rw-p 7fcc93da3000 00:00 0 348160 : 7fcc93dd1000-7fcc93e26000 r-xp 00000000 08:03 960821 + /lib64/libm-2.9.so 2093056 : 7fcc93e26000-7fcc94025000 ---p 00055000 08:03 960821 + /lib64/libm-2.9.so 4096 : 7fcc94025000-7fcc94026000 r--p 00054000 08:03 960821 + /lib64/libm-2.9.so 4096 : 7fcc94026000-7fcc94027000 rw-p 00055000 08:03 960821 + /lib64/libm-2.9.so 8192 : 7fcc94027000-7fcc94029000 r-xp 00000000 08:03 960819 + /lib64/libdl-2.9.so 2097152 : 7fcc94029000-7fcc94229000 ---p 00002000 08:03 960819 + /lib64/libdl-2.9.so 4096 : 7fcc94229000-7fcc9422a000 r--p 00002000 08:03 960819 + /lib64/libdl-2.9.so 4096 : 7fcc9422a000-7fcc9422b000 rw-p 00003000 08:03 960819 + /lib64/libdl-2.9.so 86016 : 7fcc9422b000-7fcc94240000 r-xp 00000000 08:03 960824 + /lib64/libnsl-2.9.so 2093056 : 7fcc94240000-7fcc9443f000 ---p 00015000 08:03 960824 + /lib64/libnsl-2.9.so 4096 : 7fcc9443f000-7fcc94440000 r--p 00014000 08:03 960824 + /lib64/libnsl-2.9.so 4096 : 7fcc94440000-7fcc94441000 rw-p 00015000 08:03 960824 + /lib64/libnsl-2.9.so 8192 : 7fcc94441000-7fcc94443000 rw-p 7fcc94441000 00:00 0 1253376 : 7fcc94443000-7fcc94575000 r-xp 00000000 08:03 906609 + /usr/lib64/libdb-4.5.so 2097152 : 7fcc94575000-7fcc94775000 ---p 00132000 08:03 906609 + /usr/lib64/libdb-4.5.so 12288 : 7fcc94775000-7fcc94778000 r--p 00132000 08:03 906609 + /usr/lib64/libdb-4.5.so 12288 : 7fcc94778000-7fcc9477b000 rw-p 00135000 08:03 906609 + /usr/lib64/libdb-4.5.so 122880 : 7fcc9477b000-7fcc94799000 r-xp 00000000 08:03 960806 + /lib64/ld-2.9.so 28672 : 7fcc94922000-7fcc94929000 r--s 00000000 08:03 919953 + /usr/lib64/gconv/gconv-modules.cache 225280 : 7fcc94929000-7fcc94960000 r--p 00000000 08:03 1067394 + /usr/lib/locale/de_DE/LC_CTYPE 16384 : 7fcc94960000-7fcc94964000 rw-p 7fcc94960000 00:00 0 8192 : 7fcc94996000-7fcc94998000 rw-p 7fcc94996000 00:00 0 4096 : 7fcc94998000-7fcc94999000 r--p 0001d000 08:03 960806 + /lib64/ld-2.9.so 4096 : 7fcc94999000-7fcc9499a000 rw-p 0001e000 08:03 960806 + /lib64/ld-2.9.so 86016 : 7fffb8343000-7fffb8358000 rw-p 7ffffffe9000 00:00 0 + [stack] 4096 : 7fffb83b4000-7fffb83b5000 r-xp 7fffb83b4000 00:00 0 + [vdso] 4096 : ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 + [vsyscall] total: 22032k [download] This should correspond to the VSZ as reported by `ps`: `$ /usr/bin/perl -e 'system "ps -p $$ -o rss,vsz,comm"' RSS VSZ COMMAND 1796 22032 perl` [download] As you can see, it's mostly mappings to libraries. But as I already mentioned, a virtual memory mapping doesn't mean it's also using real physical memory... See also `mmap(2)`. And maybe Understanding Memory and Understanding The Linux Virtual Memory Manager (PDF, 730p.) for further reading (I just had those two links within reach — Google's (or DDG's) index knows more fine documents on the subject :)	[reply] [d/l] [select]
Re^4: memory usage on x86_64 by powerman (Friar) on Feb 16, 2012 at 11:00 UTC
Re: memory usage on x86_64 by BrowserUk (Patriarch) on Feb 15, 2012 at 20:00 UTC
FYI: On Win, the baseline memory usage seems to have dropped between 5.8.9 & 5.10.1: c:\test>\perl32\bin\perl -mversion -e"print $version::VERSION; system +qq[tasklist /fi \"pid eq $$\"]" 0.76 Image Name PID Session Name Session# Mem + Usage ========================= ======== ================ =========== ====== +====== perl.exe 26008 Console 1 7 +,712 K c:\test>\perl64\bin\perl -mversion -E"say $version::VERSION; system qq +[tasklist /fi \"pid eq $$\"]" 0.78 Image Name PID Session Name Session# Mem + Usage ========================= ======== ================ =========== ====== +====== perl.exe 42052 Console 1 6 +,524 K [download] Are you sure your perls are built with similar build options? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l]
Re^2: memory usage on x86_64 by powerman (Friar) on Feb 15, 2012 at 21:19 UTC
Yes, I'm sure. It's Gentoo - I've compiled perl from source with exactly same options and environment libraries on both systems. Only difference is kernel version because I didn't upgraded kernel on one of systems yet (will do this tomorrow), but I don't believe kernel version can make any difference here. So, both perls are same, and only difference is 32/64-bit system.	[reply]
Re^3: memory usage on x86_64 by BrowserUk (Patriarch) on Feb 15, 2012 at 21:46 UTC
The only thing that came to mind that might account for the difference was if something was being statically rather than dynamically linked. Could that still apply to one of the indirect components? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^4: memory usage on x86_64 by powerman (Friar) on Feb 16, 2012 at 08:10 UTC