Plankton has asked for the wisdom of the Perl Monks concerning the following question:

Dear Wise Monks,

I am testing changes to my 

limit.conf
on my RHEL4 system. I want to see what happens if I set my virtual memory very low ... like ... 

$ ulimit -a
...
virtual memory        (kbytes, -v) 8192

[download]

... and then I run Perl script that fork's off some memory draining processes ... like so ... 

$ cat swaphogger.pl
#!/usr/bin/perl -w

use strict;

sub start_hogging_memory  {
        my @blackhole;
        my $count = 0;
        print __FILE__ . " [" . __LINE__ . "] child [$$]\n";
        open ( FH, "> ${0}_${$}.output" ) or die " could not open file
+:$!\n";
        my $swapon = `swapon -s | tail -1`;
        while ( 1 ) { 
                chomp( $swapon );
                print FH "[$swapon $count]\n"; 
                push @blackhole, "$count -----------------------------
+------------------------------------------\n"; 
                $count++;
                $swapon = `swapon -s | tail -1` # strangeness
        }
        close (FH);
}

my $maxChildren = shift;
my $numChildren = 0;

while ( $numChildren < $maxChildren ) {
        $numChildren++;
        my $pid = fork();
        if ( $pid == 0 ) {
                start_hogging_memory ();
        } elsif ( $pid ) {
                print __FILE__ . " [" . __LINE__ . "] parent [$$]\n";
        } else {
                die __FILE__ . " [" . __LINE__ . "] couldn't fork() :$
+!\n";
        }
}

[download]

The funny thing is that above code does NOT use up any swap! But if I comment out the line : 

                $swapon = `swapon -s | tail -1` # strangeness

[download]

... things work the way I would expect .... 

$ ./swaphogger.pl  10
./swaphogger.pl [8] child [2583]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2584]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2585]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2588]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2589]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2590]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2591]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2592]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2593]
./swaphogger.pl [30] parent [2582]
./swaphogger.pl [8] child [2594]
./swaphogger.pl [30] parent [2582]
rcecala@eman-sjc1-001: Priority=4 scripts$ Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!
Out of memory!

[download]

Why does the line ... 

                $swapon = `swapon -s | tail -1` # strangeness

[download]
... seem to fix my memory hole?

Replies are listed 'Best First'.
Re: Swap hog - I do not understand this behavoir
by BrowserUk (Patriarch) on Apr 23, 2009 at 20:20 UTC

    Could it be that you just didn't wait long enough?

    Spawning two processes (swapon & tail), each time around in your inner loop is going to slow thing down 100 or maybe 1000 fold.

    Try modifying your loop to only spawn the processes much less frequently and it'll probably move into swaping much more quickly: 

            my $count = 0;
        while ( 1 ) { 
                chomp( $swapon );
                print FH "[$swapon $count]\n"; 
                push @blackhole, "$count -----------------------------
+----------
+--------------------------------\n"; 
                $count++;
                $swapon = `swapon -s | tail -1` unless ++$count % 1000
+;
        }

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
Re: Swap hog - I do not understand this behavoir
by tilly (Archbishop) on Apr 23, 2009 at 20:26 UTC
    Is it possible that the swapon command is slow enough that your large processes don't get large as quickly as you expect?
[reply]
Re: Swap hog - I do not understand this behavoir
by ig (Vicar) on Apr 24, 2009 at 04:10 UTC
    The funny thing is that above code does NOT use up any swap!

    Your system will only use swap space if system wide memory utilization approaches or exceeds available physical memory. With per process virtual memory limited to 8192KB, each of your test processes can consume no more than this amount for the process itself and some additional amount for file buffers. You can see your total system memory and memory utilization with 'top' or 'vmstat'. Even if you have only 512MB on your system it will take more than 10 of your test processes to consume it all (10 processes would probably consume less than 100MB total).

    As for your test processes running out of virtual memory: they are consuming about 100 bytes on each loop iteration (I don't know the exact amount but there is some overhead on the ~80 character string you are pushing onto your array in each iteration). It takes over 20,000 iterations to reach a total process memory utilization of 8192KB (22786 on my system) and, as others have pointed out, with a process being spawned and file I/O in each iteration, it takes considerable time to reach this number of iterations.

    If you start your test and tail your output files, you should see them continue to grow until the processes fail. The child processes will continue after your main process has exited. Just be patient and eventually you should see an 'Out of memory!' for each child process.

[reply]

Back to Seekers of Perl Wisdom