Perl_Love has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a few programs, do not know why the memory has been increased from 20M has been added to xxG.

The number of processes to keep about 100.

Please help me, thank you!

My environment:

CentOS7

Perl 5.20.3

#!/usr/bin/perl -w # fork.pl $SIG{INT}=\&INT_EXIT; use IO::Socket::SSL; use Mojo::Asset; use Mojo::Asset::File; use Mojo::Asset::Memory; use Mojo::Base; use Mojo::Content; use Mojo::Content::MultiPart; use Mojo::Content::Single; use Mojo::Cookie; use Mojo::Cookie::Request; use Mojo::EventEmitter; use Mojo::Exception; use Mojo::IOLoop; use Mojo::IOLoop::Client; use Mojo::IOLoop::Delay; use Mojo::IOLoop::Server; use Mojo::IOLoop::Stream; use Mojo::JSON; use Mojo::Loader; use Mojo::Message; use Mojo::Parameters; use Mojo::Path; use Mojo::Reactor; use Mojo::Reactor::Poll; use Mojo::Server; use Mojo::Server::Daemon; use Mojo::Transaction; use Mojo::Transaction::HTTP; use Mojo::Transaction::WebSocket; use Mojo::URL; use Mojo::UserAgent; use Mojo::UserAgent::CookieJar; use Mojo::UserAgent::Proxy; use Mojo::UserAgent::Server; use Mojo::UserAgent::Transactor; use Mojo::Util; use Mojo::WebSocket; use File::Find; use Parallel::ForkManager; $|=1; my %ID; my $dir='/home/root/Perl/zhubo/AccountID'; my $fork; my %Header; my $ua; &Config; &Header; &Mojo_UA; my $pm=new Parallel::ForkManager($fork); sub Config{ open(F,"conf.json") or die; my @conf=<F>; close F; chomp @conf; my $json=join('',@conf); $json_conf=Mojo::JSON::decode_json($json); $fork=$json_conf->{'fork'}; } sub Header{ my @header=<headers/*>; foreach my $H(@header){ next unless($H=~m/\.header$/); my $hn; $hn=$1 if($H=~m/.*\/(.*?)\.header$/); open(H,$H) or die; while(my $h=<H>){ chomp $h; my @H=split('=>',$h); $Header{$hn}{'header'}{$H[0]}=$H[1]; } close H; } } sub Mojo_UA{ $ua=Mojo::UserAgent->new; $ua=$ua->connect_timeout($json_conf->{'connect_timeout'}); $ua=$ua->inactivity_timeout($json_conf->{'inactivity_timeout'}); $ua=$ua->max_redirects($json_conf->{'max_redirects'}); } while(1){ File::Find::find(\&wanted,$dir); } $pm->wait_all_children; sub wanted { if(-f $File::Find::name){ if($File::Find::name=~m/\.next$/){ if(exists $ID{$_}){ my $o=$File::Find::name; $o=~s/\.next$/.bak/; unlink($o); $o=~s/\.bak$//; unlink($o); delete $ID{$_}; } } if($File::Find::name=~m/\.bak$/){ my $gh=$_; my $e=$gh; $gh=~s/\.bak$//; my $f=$_; $f=~s/\.bak$//; unless(exists $ID{$f}){ my $fuck=&AccountID($f); if($fuck ne 'NULL'){ $ID{$f}=1; my $pid=$pm->start and next; system("/home/root/Perl/zhubo/ZhuBo.pl $f"); delete $ID{$f}; rename($e,$gh); $pm->finish; } undef $fuck; } } else{ my $r=$File::Find::name.'.bak'; rename($File::Find::name,$r); unless(exists $ID{$_}){ my $fuck=&AccountID($_); if($fuck ne 'NULL'){ $ID{$_}=1; my $pid=$pm->start and next; system("/home/root/Perl/zhubo/ZhuBo.pl $_"); my $e=$r; $e=~s/\.bak$//; delete $ID{$_}; rename($r,$e); $pm->finish; } undef $fuck; } } } } sub AccountID{ my $accountId=shift; my $url='https://taobaolive.taobao.com/room/index.htm?userId='.$acco +untId; my $res; while(1){ my $eval=eval{ $res=$ua->get($url=>{%{$Header{'item_list'}{'header'}}})->result +; }; last if($eval); } if($res->is_success){ my $body=$res->body; if($body=~m/var liveDetail \= (.*?) \|\| \{\}\;/s){ my $liveDetail=$1; unless($liveDetail=~m/\}$/){ undef $res; undef $body; undef $liveDetail; return 'NULL'; } my $json_hash=Mojo::JSON::decode_json($liveDetail); if(exists $json_hash->{'liveId'}){ if($json_hash->{'liveId'} ne '0'){ my $liveId=$json_hash->{'liveId'}; undef $json_hash; undef $res; undef $body; undef $liveDetail; return $liveId; } else{ undef $json_hash; undef $res; undef $body; undef $liveDetail; return 'NULL'; } } else{ undef $json_hash; undef $res; undef $body; undef $liveDetail; return 'NULL'; } } else{ &AccountID($accountId); } } else{ &AccountID($accountId); } } sub INT_EXIT{ exit; }

Replies are listed 'Best First'.
Re: Memory has been increasing, unable to release memory.
by Eily (Monsignor) on Mar 27, 2017 at 13:23 UTC

    There are several issues with your code:

    • You have neither used strict nor warnings
    • You have two while(1) loops, only one of which has a last statement. Your program won't terminate unless you kill it.
    • Your next statements aren't in the scope of any loop. Although next can be used to exit a sub called in a loop (and I'm quite surprised it works), I'm not sure if it is safe, and I'm certain that it is not clear. Also, warnings would have warned you about this. You can use return instead, although it looks like you just copied the example from Parallel::ForkManager without understanding it.
    • $fork is never set, so the value transmitted to "new" is undef, interpreted as 0. The documentation states "If you specify 0 (zero), then no children will be forked". So ForkManager isn't doing what you think it does.
    • You can probably remove half the use statements.
    • As far as I know, all your undef calls are useless, because the variables are implicitly freed when their scope is exited.
    • You don't have comments, and you didn't explain what your program is supposed to do.
    • One letter variables should be avoided (they can have special meanings), $fuck as well.
    Please read How do I post a question effectively?

    It looks like you want to call another perl program on a list of files. You should turn that program into a module, with one exported function, and call that function on your files, rather than spawn a new process with system. You'll find some information on modules in the perlmod documentation and in tutorials like this one

      Hi Eily . thank you for such a detailed answer!

      Actually, I wrote the two script with the fork.pl call another crawler.pl .The two script memory will keep increasing, I suspect I wrote the script with some problem, the reference count not zero, so the memory is not released.

      $fork, I placed in sub Config, $fork=$json_conf->{'fork'}, I set up 500 in the conf.json file.

      undef, I tried to release the memory, really no effect !

      Two while (1): are a crawler script, the first while (1), I let it keep running. While the second while (1), because I found that sometimes get error (it seems that SSL error or other things what I forget) the script will be directly out.So I have a idea with eval, it can work, although it is not perfect, let me change it again :)

      You suggest to changed script to modules, whether should also solved memory surge problem ,so that it will have effect ?

      I would like to use the Devel:: Peek module, but 5.20.3 version can not install, I do not know how to analyze the problem of memory surge, please help me to check Thanks a lot!

        Anonymonk's answer seems to point out a good candidate for your memory problem. My suggestions were rather means to make your code easier to read and understand, which would make debugging easier. I'm sure that creating so many independant perl processes doesn't help though. Adding use strict; and use warnings; can help you solve your memory issues though, as it points out potential mistakes and known pitfalls.

        And you're right about $fork, I did not see that it is indeed set. But this demonstrates that your code is hard to read. By the way, the & isn't necessary to call a function, instead, if you want to make a call more explicit, you can use parentheses, like this: Config();.

        You suggest to changed script to modules, whether should also solved memory surge problem ,so that it will have effect ?
        It might help with some problems, and it might make other problems worse. Unfortunately, leaking memory is one of the problems that might be made worse. In most cases, I would tend to agree with Eily's advice, but it's probably not a high priority for you.
Re: Memory has been increasing, unable to release memory.
by Anonymous Monk on Mar 27, 2017 at 15:30 UTC
    AccountID handles errors by recursing, so you might just be building up a massive call stack. You should write your code with use warnings; like Eily suggests, so you would get a message about that. Try putting a debugging print in there to see if it's fetching the same account over and over again.
    sub AccountID{ my $accountId=shift; print "Fetching account $accountId\n";
    If it turns out that's what is happening, you need to find a better way to handle failures. Don't just keep retrying them over and over again forever!
      Well, thank you. I'll try it when I'm at work.