in reply to Re: Memory has been increasing, unable to release memory.
in thread Memory has been increasing, unable to release memory.

Hi Eily . thank you for such a detailed answer!

Actually, I wrote the two script with the fork.pl call another crawler.pl .The two script memory will keep increasing, I suspect I wrote the script with some problem, the reference count not zero, so the memory is not released.

$fork, I placed in sub Config, $fork=$json_conf->{'fork'}, I set up 500 in the conf.json file.

undef, I tried to release the memory, really no effect !

Two while (1): are a crawler script, the first while (1), I let it keep running. While the second while (1), because I found that sometimes get error (it seems that SSL error or other things what I forget) the script will be directly out.So I have a idea with eval, it can work, although it is not perfect, let me change it again :)

You suggest to changed script to modules, whether should also solved memory surge problem ,so that it will have effect ?

I would like to use the Devel:: Peek module, but 5.20.3 version can not install, I do not know how to analyze the problem of memory surge, please help me to check Thanks a lot!

  • Comment on Re^2: Memory has been increasing, unable to release memory.

Replies are listed 'Best First'.
Re^3: Memory has been increasing, unable to release memory.
by Eily (Monsignor) on Mar 27, 2017 at 16:00 UTC

    Anonymonk's answer seems to point out a good candidate for your memory problem. My suggestions were rather means to make your code easier to read and understand, which would make debugging easier. I'm sure that creating so many independant perl processes doesn't help though. Adding use strict; and use warnings; can help you solve your memory issues though, as it points out potential mistakes and known pitfalls.

    And you're right about $fork, I did not see that it is indeed set. But this demonstrates that your code is hard to read. By the way, the & isn't necessary to call a function, instead, if you want to make a call more explicit, you can use parentheses, like this: Config();.

[reply]
[d/l]
[select]

      Hi~

      My other question:

      http://www.perlmonks.org/?node_id=1165988

      afoken reply about strict, I add use strict now.

      $fork.pl set max 500 fork, and all file is 300.

      Half the file called crawler.pl, and the other half of the file has been running in fork.pl.

      print account, I can't find anything problem

      Maybe I need to rewrite the whole program

[reply]
Re^3: Memory has been increasing, unable to release memory.
by Anonymous Monk on Mar 27, 2017 at 17:08 UTC
    You suggest to changed script to modules, whether should also solved memory surge problem ,so that it will have effect ?
    It might help with some problems, and it might make other problems worse. Unfortunately, leaking memory is one of the problems that might be made worse. In most cases, I would tend to agree with Eily's advice, but it's probably not a high priority for you.
[reply]

      delete system("/home/root/Perl/zhubo/crawler.pl $_");

      add sleep 11111;

      Problems exist

[reply]
        11111 seconds is just over 3 hours. I'm not sure why you want to sleep that long.
[reply]

In Section Seekers of Perl Wisdom