mtmcc has asked for the wisdom of the Perl Monks concerning the following question:

Worthy Monks,

I have a few perl scripts that manipulate some text in large text files (~20,000,000 lines), using while (<FH>) loops to run through the file. Apart from a few small arrays and hashes, I use very little memory.

The programs work just as expected both on Ubuntu 12.04 (Perl 5.14.2), and OSX mountain lion (Perl 5.12.4).

However, in OSX, the available 'free' memory is quickly converted to 'inactive' memory. The system monitor on Ubuntu shows pretty much flat memory usage.

The only downside of this, as far as I can see, is that in OSX, other programs running simultaneously are slower to respond. My script seems to be unaffected.

I was wondering if anybody else has noticed this, and is there an obvious general explanation that I'm missing?

Thank you for any opinions you might have.

PS As an example, used on a text file with 20,000,000 lines: 

#!/usr/bin/perl 
use strict;

my $fileForLineCount = $ARGV[0];
my $lineCount = 0;
my $buffer = 0;

open (my $targetHandle, "<", $fileForLineCount);
while(sysread $targetHandle, $buffer, 4096)
    {
        $lineCount += ($buffer =~ tr/\n//);
    }

close $targetHandle;

print STDERR "$lineCount\n\n";

[download]
on Ubuntu, it flies through, and on OSX, the memory thing happens.

Thanks for your help!

Replies are listed 'Best First'.
Re: Inactive memory and OSX
by BrowserUk (Patriarch) on Jul 04, 2013 at 22:01 UTC

    See http://apple.stackexchange.com/questions/67031/isnt-inactive-memory-a-waste-of-resources. (And then look for the purge command.)

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Inactive memory and OSX
by dave_the_m (Monsignor) on Jul 04, 2013 at 21:41 UTC
    Not knowing the definition of 'inactive memory' in OSX, I can only speculate, but...

    Perhaps its just memory being used as buffer cache by the OS when reading large (GB+) sized files. Does something like cat bigfile | grep NoSuchPattern show similar OS memory usage?

    Dave.

[reply]
[d/l]
      OS X seems to use a similar reporting scheme as FreeBSD (documented here):
      Mem: 346M Active, 319M Inact, 487M Wired, 4584K Cache, 392M Buf, 2530M Free

      Inactive in OS X is a code word for the file system cache; hence, nothing to worry about (even though a fair amount of people foolishly do)

[reply]
[d/l]
Re: Inactive memory and OSX
by mtmcc (Hermit) on Jul 05, 2013 at 07:48 UTC
    Thank you both!

    Dave, you're absolutely right, cat | grep does the same thing...

    ...and thanks for the link Browser, it makes more sense now. I had been purging after each run. It's a little irritating, because others will likely need to use these programs in future, and some of them may use OSX.

    It would be ideal if it were possible to "switch off" this behaviour from start of the script.

    For now, purging at the end will have to do:

     system("purge") if $^O eq "darwin";

    Thanks guys!
[reply]
[d/l]
      It would be ideal if it were possible to "switch off" this behaviour from start of the script. For now, purging at the end will have to do:

      The way I understand it, inactive memory gets returned to the system memory pool on demand. That is, as soon as something else (another process) needs it, it gets made available immediately and cheaply without any need to manually purge it.

      It is only held as 'inactive' while nothing else asks for it, on the off chance (and frequently true), that you might re-run the same program and thus it makes the second (and any subsequent runs) of the program quicker.

      In that respect, you can simple consider inactive memory a part of the free memory pool and your 'need to purge' is purely psychological.

      I'd also have questions regarding whether running purge from inside the process that you're trying to purge will work. system starts another process, but your program won't finish until the system call returns. (I can't try it; its my brother that runs OSX and its the middle of the night where he is.)

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        your 'need to purge' is purely psychological

        You're probably right! It does seem to me that browsers and text editors etc are slower to load and run while this is going on, but maybe that's simply because the perl script claims the 'inactive' memory that these other programs had been using.

        The system("purge") call does work, but as you point out, it's probably just as well to leave it out.

        Thanks again!

        Michael
[reply]

Back to Seekers of Perl Wisdom