rmahin has asked for the wisdom of the Perl Monks concerning the following question:

SOLVED Updated to perl 5.20.2 and all problems vanished.

Hello! Been noticing some strange memory issues with a script I'm working on, and have come up with this to highlight the issues

use strict;
use warnings;
use threads;
use threads::shared;
$|++;

my $DONE :shared = 0;
my $lock :shared;

my $execMethod = $ARGV[0] || 0;
if($execMethod !~ /[12345]/){
    print "Must pass an exec method:\n";
    print "1 = backticks\n";
    print "2 = backticks synchronized\n";
    print "3 = open\n";
    print "4 = open synchronized\n";
    print "5 = system\n";
    exit 1;
}
sub execute{
    my $cmd = shift;
    if($execMethod == 1){
        `$cmd`
    }elsif($execMethod == 2){
        lock $lock;
        `$cmd`
    }elsif($execMethod == 3){
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
    }elsif($execMethod == 4){
        lock $lock;
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
    }elsif($execMethod == 5){
        system($cmd . ">nul");
    }
}

sub worker{
    while(!$DONE){
        execute('echo hello world');
    }
}
my @workers = map threads->create( \&worker), (1..30);
print "Press <enter> to terminate\n";
<STDIN>;
$DONE = 1;

$_->join() for @workers;

[download]

Script is invoked with <script>.pl executionMethod

Question 1 relates to memory usage. Methods 2, 4, and 5 all exhibit a permanent memory creep. Is there a way to fix this and let this script run forever? If the solution is periodically join threads, this is not feasible. I have tried this a couple years ago and came across some bug but outlined in this node and changing to an approach like this would require far too much testing to ensure no other bugs occur. If there really is no solution, would be good to know :)

Question 2: For methods 1 and 3, why do i need to synchronize them? If I do not, the script just hangs up usually after only a few seconds. Synchronizing this access kind of seems to defeat the purpose of running system commands in threads, is there something I can do differently?

*I have been running this on Windows 2008 R2, using perl 5.18.2

Thanks in advance for all your help

UPDATE 1: Here is the log file from perfmon running test 5 for a little over a minute and eating roughly 150MB.

UPDATE 2:

perl -v output

This is perl 5, version 18, subversion 1 (v5.18.1) built for MSWin32-x
+64-multi-thread
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2013, Larry Wall

Binary build 1800 [297570] provided by ActiveState http://www.ActiveSt
+ate.com
Built Sep 20 2013 15:07:17

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using "man perl" or "perldoc perl".  If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.

[download]

systeminfo

Host Name:                 VCLOUD291
OS Name:                   Microsoft Windows Server 2008 R2 Enterprise
OS Version:                6.1.7601 Service Pack 1 Build 7601
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                55041-507-5915375-84291
Original Install Date:     4/23/2014, 12:10:03 PM
System Boot Time:          7/31/2015, 1:49:06 PM
System Manufacturer:       Microsoft Corporation
System Model:              Virtual Machine
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 46 Stepping 6 
+GenuineIntel ~1862 M
hz
BIOS Version:              American Megatrends Inc. 090004 , 3/19/2009
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-07:00) Arizona
Total Physical Memory:     4,096 MB
Available Physical Memory: 1,635 MB
Virtual Memory: Max Size:  6,143 MB
Virtual Memory: Available: 3,364 MB
Virtual Memory: In Use:    2,779 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    WORKGROUP
Logon Server:              \\VCLOUD291
Hotfix(s):                 46 Hotfix(s) Installed.

[download]

Replies are listed 'Best First'.
Re: Perl system command memory usage in threads
by BrowserUk (Patriarch) on Jul 31, 2015 at 22:49 UTC
    Question 2: For methods 1 and 3, why do i need to synchronize them? If I do not, the script just hangs up usually after only a few seconds.

    I can confirm this problem. I believe it to be a bug in Perl. I think it is to do with the implementation of the wait/waitpid emulation on windows which limits the number of concurrent child processes to 64. I'll try to tie that down a bit further tomorrow and raise a perlbug for it if I succeed.

    Meanwhile, on the basis of what I've seen of your code, you should probably try MCE for this task.

    Anyone got any experience of this phone's predecessor?

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
[reply]

      Awesome, glad this one wasn't just me! Again, this problem also seemed to clear up int 5.20, so maybe its been fixed? Or at least masked a bit better :P

      That MCE sounds really intriguing. In the full scale version of my code I make use of Thread::Queue and a pool threads (as you have taught me over the years), in your experience how large are the performance/reliability benefits of using MCE over such a model? Thanks!

[reply]
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Perl system command memory usage in threads
by BrowserUk (Patriarch) on Jul 31, 2015 at 19:14 UTC
[reply]

      See Update 1, if I'm doing something wrong in my data collection, or you need additional info please let me know!

[reply]
Re: Perl system command memory usage in threads
by marioroy (Prior) on Aug 01, 2015 at 18:47 UTC

    Hi all.

    The following is a modified version using MCE from trunk. Thus, will work when MCE 1.7 is released. All 5 exec methods ran without the memory creeping issues with or without threads.

    The Perl version is 5.16.2 on Mac OS X 10.9.5. Also, tested on a CentOS 7 VM with Perl v5.16.3 without any problems. Finally, tested on a Windows 7 VM with Strawberry Perl v5.16.3.1 and v5.18.4.1. No problems with either.

    use strict;
use warnings;
use threads;       # using threads is optional

use MCE::Mutex;
use MCE::Flow;
use MCE::Shared;

my $DONE : Shared = 0;
my $lock = new MCE::Mutex;

my $execMethod = $ARGV[0] || 0;
if($execMethod !~ /[12345]/){
    print "Must pass an exec method:\n";
    print "1 = backticks\n";
    print "2 = backticks synchronized\n";
    print "3 = open\n";
    print "4 = open synchronized\n";
    print "5 = system\n";
    exit 1;
}
sub execute{
    my $cmd = shift;
    if($execMethod == 1){
        `$cmd`;
    }elsif($execMethod == 2){
        $lock->lock;
        `$cmd`;
        $lock->unlock;
    }elsif($execMethod == 3){
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
    }elsif($execMethod == 4){
        $lock->lock;
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
        $lock->unlock;
    }elsif($execMethod == 5){
        system($cmd . ">nul");
    }
}

sub worker{
    while(!$DONE){
        execute('echo hello world');
    }
}

sub console{
    print "Press <enter> to terminate\n";
    <STDIN>;
    $DONE = 1;
}

# two subtasks; 30 workers, and 1 for console
mce_flow { max_workers => [ 30, 1 ] }, \&worker, \&console;

    [download]

    The OP's demonstration is possible with MCE. It requires 1.7 which will be released soon.

    Kind regards, Mario.

[reply]
[d/l]
Re: Perl system command memory usage in threads
by Anonymous Monk on Jul 31, 2015 at 20:15 UTC
    Give us some numbers here ... how much memory, how much creep, over how much time, and what makes you call it permanent?
[reply]

      See Update 1. The rate of growth shown there stays the same throughout the life cycle of the script (which is what I mean by permanent).

      If you need anything else let me know!

[reply]

        Is that the exact code from the OP?

        Because when I run that code (Vista;(v5.18.4) built for MSWin32-x64-multi-thread) with:

        • Option 2: 22.4MB constant regardless of how long I run it.
        • Option 4: 22.3MB.
        • Option 5: 23.3MB.

        Which brand (AS/Strawberry/self-built) of Perl are you using?

        Anyone got any experience of this phone's predecessor?

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.
        I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
[reply]

Back to Seekers of Perl Wisdom