in reply to memory usage on x86_64

"softlimit -m" also limits the maximum virtual address space (i.e. it calls setrlimit(2) on the resource RLIMIT_AS1) — see the man page for details).   So what you're testing here is the virtual memory usage.

The virtual memory usage of a program, however, is not necessarily a useful measure of how much memory a program actually currently "needs".  It's more a measure of the address space that the program can potentially address (without allocating more), not of how much physical memory it actually has in use — the resident set size (RSS) is a better measure of the latter.   In other words, I wouldn't be worried too much :)

For comparison, my 64-bit Perl 5.12.4 on Ubuntu uses: 

$ perl -le 'use version; print $version::VERSION; system "ps -p $$ -o 
+rss,vsz,comm"'
0.82
  RSS    VSZ COMMAND
 2476  24236 perl

[download]

___

1) 

$ strace -esetrlimit softlimit -m 25000000 perl -e1
setrlimit(RLIMIT_DATA, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) = 
+0
setrlimit(RLIMIT_STACK, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) =
+ 0
setrlimit(RLIMIT_MEMLOCK, {rlim_cur=64*1024, rlim_max=64*1024}) = 0
setrlimit(RLIMIT_AS, {rlim_cur=25000000, rlim_max=RLIM_INFINITY}) = 0

[download]

Replies are listed 'Best First'.
Re^2: memory usage on x86_64
by locked_user sundialsvc4 (Abbot) on Feb 15, 2012 at 23:56 UTC

    IMnsHO, this is a crucial observation.   The virtual memory manager is always “lazy.”   If the memory is available, and if there is no “pressure” actually being exerted upon the virtual-memory resource, then the VMM will make absolutely no effort to clean up.   After all, it is perfectly plausible that, someday, you might ask for that same bit of information again.   It is to the system’s advantage to presume that you will do so.

    VMM’s are designed to respond only to actual memory pressure.   Like a typical bachelor, it won’t load the dishwasher until three minutes before its girlfriend is due to knock on the door.   And, on our typically fast and typically capacious “developer” machines, that rarely happens.   (Memory pressure, I mean ... not girlfriends.   We hope.)   Consequently, memory requirements are customarily reported too large.

Re^2: memory usage on x86_64
by powerman (Friar) on Feb 16, 2012 at 07:56 UTC
    You right about RSS/VSZ: 
    ### x86
# perl -le 'use version; print $version::VERSION; system "ps -p $$ -o 
+rss,vsz,comm"'
0.96
  RSS    VSZ COMMAND
 2068   5552 perl
### x86_64
# perl -le 'use version; print $version::VERSION; system "ps -p $$ -o 
+rss,vsz,comm"'
0.96
  RSS    VSZ COMMAND
 2468  20044 perl

    [download]
    But thing is, virtual or resident, memory limits should be significantly increased on 64-bit systems. Real issue I had because of this was related to qmail (which by default run under setlimit) filter - perl script which implement greylisting.

    As you see, when perl doesn't get enough memory it's error messages ain't very informative. So, that I got in system logs is useless messages about my locale, coming from unknown source - and at that time my system rejects incoming emails because part of qmail execution chain was crashed.

    And it's still doesn't clear for me, is so much virtual memory requested by perl on 64-bit system is a bug in perl, or correct behaviour.

[reply]
[d/l]
      And it's still doesn't clear for me, is so much virtual memory requested by perl on 64-bit system is a bug in perl, or correct behaviour.

      If you have doubts things are as they should be, you can investigate the virtual memory mappings of the process via /proc/<PID>/maps  (or /proc/self/maps from within the process).  See also proc(5) for a short explanation of the columns.

      The following snippet might help with computing sizes: 

      #!/usr/bin/perl

open my $maps, "<", "/proc/self/maps" or die $!;

my $sum;
while (<$maps>) {
    /^(\S+)-(\S+)/;
    my $size = hex($2)-hex($1);
    $sum += $size;
    printf "%8d : %s", $size, $_;
}
printf "total: %dk\n", $sum/1024;

      [download]

      On my current system (64-bit openSUSE 11.1, Perl 5.12.3), it shows 

       1454080 : 00400000-00563000 r-xp 00000000 08:03 1209182              
+              /usr/local/perl/5.12.3/bin/perl
    4096 : 00762000-00763000 r--p 00162000 08:03 1209182              
+              /usr/local/perl/5.12.3/bin/perl
   16384 : 00763000-00767000 rw-p 00163000 08:03 1209182              
+              /usr/local/perl/5.12.3/bin/perl
  270336 : 00767000-007a9000 rw-p 00767000 00:00 0                    
+              [heap]
 1372160 : 7fcc9341c000-7fcc9356b000 r-xp 00000000 08:03 960813       
+              /lib64/libc-2.9.so
 2097152 : 7fcc9356b000-7fcc9376b000 ---p 0014f000 08:03 960813       
+              /lib64/libc-2.9.so
   16384 : 7fcc9376b000-7fcc9376f000 r--p 0014f000 08:03 960813       
+              /lib64/libc-2.9.so
    4096 : 7fcc9376f000-7fcc93770000 rw-p 00153000 08:03 960813       
+              /lib64/libc-2.9.so
   20480 : 7fcc93770000-7fcc93775000 rw-p 7fcc93770000 00:00 0 
   90112 : 7fcc93775000-7fcc9378b000 r-xp 00000000 08:03 960839       
+              /lib64/libpthread-2.9.so
 2097152 : 7fcc9378b000-7fcc9398b000 ---p 00016000 08:03 960839       
+              /lib64/libpthread-2.9.so
    4096 : 7fcc9398b000-7fcc9398c000 r--p 00016000 08:03 960839       
+              /lib64/libpthread-2.9.so
    4096 : 7fcc9398c000-7fcc9398d000 rw-p 00017000 08:03 960839       
+              /lib64/libpthread-2.9.so
   16384 : 7fcc9398d000-7fcc93991000 rw-p 7fcc9398d000 00:00 0 
    8192 : 7fcc93991000-7fcc93993000 r-xp 00000000 08:03 960847       
+              /lib64/libutil-2.9.so
 2093056 : 7fcc93993000-7fcc93b92000 ---p 00002000 08:03 960847       
+              /lib64/libutil-2.9.so
    4096 : 7fcc93b92000-7fcc93b93000 r--p 00001000 08:03 960847       
+              /lib64/libutil-2.9.so
    4096 : 7fcc93b93000-7fcc93b94000 rw-p 00002000 08:03 960847       
+              /lib64/libutil-2.9.so
   53248 : 7fcc93b94000-7fcc93ba1000 r-xp 00000000 08:03 960817       
+              /lib64/libcrypt-2.9.so
 2097152 : 7fcc93ba1000-7fcc93da1000 ---p 0000d000 08:03 960817       
+              /lib64/libcrypt-2.9.so
    4096 : 7fcc93da1000-7fcc93da2000 r--p 0000d000 08:03 960817       
+              /lib64/libcrypt-2.9.so
    4096 : 7fcc93da2000-7fcc93da3000 rw-p 0000e000 08:03 960817       
+              /lib64/libcrypt-2.9.so
  188416 : 7fcc93da3000-7fcc93dd1000 rw-p 7fcc93da3000 00:00 0 
  348160 : 7fcc93dd1000-7fcc93e26000 r-xp 00000000 08:03 960821       
+              /lib64/libm-2.9.so
 2093056 : 7fcc93e26000-7fcc94025000 ---p 00055000 08:03 960821       
+              /lib64/libm-2.9.so
    4096 : 7fcc94025000-7fcc94026000 r--p 00054000 08:03 960821       
+              /lib64/libm-2.9.so
    4096 : 7fcc94026000-7fcc94027000 rw-p 00055000 08:03 960821       
+              /lib64/libm-2.9.so
    8192 : 7fcc94027000-7fcc94029000 r-xp 00000000 08:03 960819       
+              /lib64/libdl-2.9.so
 2097152 : 7fcc94029000-7fcc94229000 ---p 00002000 08:03 960819       
+              /lib64/libdl-2.9.so
    4096 : 7fcc94229000-7fcc9422a000 r--p 00002000 08:03 960819       
+              /lib64/libdl-2.9.so
    4096 : 7fcc9422a000-7fcc9422b000 rw-p 00003000 08:03 960819       
+              /lib64/libdl-2.9.so
   86016 : 7fcc9422b000-7fcc94240000 r-xp 00000000 08:03 960824       
+              /lib64/libnsl-2.9.so
 2093056 : 7fcc94240000-7fcc9443f000 ---p 00015000 08:03 960824       
+              /lib64/libnsl-2.9.so
    4096 : 7fcc9443f000-7fcc94440000 r--p 00014000 08:03 960824       
+              /lib64/libnsl-2.9.so
    4096 : 7fcc94440000-7fcc94441000 rw-p 00015000 08:03 960824       
+              /lib64/libnsl-2.9.so
    8192 : 7fcc94441000-7fcc94443000 rw-p 7fcc94441000 00:00 0 
 1253376 : 7fcc94443000-7fcc94575000 r-xp 00000000 08:03 906609       
+              /usr/lib64/libdb-4.5.so
 2097152 : 7fcc94575000-7fcc94775000 ---p 00132000 08:03 906609       
+              /usr/lib64/libdb-4.5.so
   12288 : 7fcc94775000-7fcc94778000 r--p 00132000 08:03 906609       
+              /usr/lib64/libdb-4.5.so
   12288 : 7fcc94778000-7fcc9477b000 rw-p 00135000 08:03 906609       
+              /usr/lib64/libdb-4.5.so
  122880 : 7fcc9477b000-7fcc94799000 r-xp 00000000 08:03 960806       
+              /lib64/ld-2.9.so
   28672 : 7fcc94922000-7fcc94929000 r--s 00000000 08:03 919953       
+              /usr/lib64/gconv/gconv-modules.cache
  225280 : 7fcc94929000-7fcc94960000 r--p 00000000 08:03 1067394      
+              /usr/lib/locale/de_DE/LC_CTYPE
   16384 : 7fcc94960000-7fcc94964000 rw-p 7fcc94960000 00:00 0 
    8192 : 7fcc94996000-7fcc94998000 rw-p 7fcc94996000 00:00 0 
    4096 : 7fcc94998000-7fcc94999000 r--p 0001d000 08:03 960806       
+              /lib64/ld-2.9.so
    4096 : 7fcc94999000-7fcc9499a000 rw-p 0001e000 08:03 960806       
+              /lib64/ld-2.9.so
   86016 : 7fffb8343000-7fffb8358000 rw-p 7ffffffe9000 00:00 0        
+              [stack]
    4096 : 7fffb83b4000-7fffb83b5000 r-xp 7fffb83b4000 00:00 0        
+              [vdso]
    4096 : ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0    
+              [vsyscall]
total: 22032k

      [download]

      This should correspond to the VSZ as reported by ps: 

      $ /usr/bin/perl -e 'system "ps -p $$ -o rss,vsz,comm"'
  RSS    VSZ COMMAND
 1796  22032 perl

      [download]

      As you can see, it's mostly mappings to libraries.  But as I already mentioned, a virtual memory mapping doesn't mean it's also using real physical memory...

      See also mmap(2).  And maybe Understanding Memory and Understanding The Linux Virtual Memory Manager (PDF, 730p.) for further reading  (I just had those two links within reach — Google's (or DDG's) index knows more fine documents on the subject :)

[reply]
[d/l]
[select]
        Thanks. Looks like on 64-bit there is extra line for each library with flags "---p" using each about 2MB, 5 such libraries add about 10MB. So, looks like this issue have nothing with perl itself. Anyway, do you know what these extra lines mean?
[reply]

In Section Seekers of Perl Wisdom