andreidf has asked for the wisdom of the Perl Monks concerning the following question:

hello,

I've searched a lot but I couldn't find a really similar thread. Please give me a pointer where to find this info or where to ask/report a bug.

Below you'll see two similar versions. First one uses lots of memory; after every call of copy function it aquires more memory and it seems perl won't release it although I do a exit & waitpid. The other strangely uses a lot less memory (it still seems to be eatting memory but no that much).

I run this scripts with perl v5.8.8 on Windows XP. (Binary build 817, ActiveState, 20 Mar 2006). I've run the first script (the one without use Win32) on Linux and it seems to work just fine (I guess this could be a platform implementation problem). Changing copy() with Win32::FileCopy() gives no real benefit (it still seems to eat up memory (slowly, but it's there)).

Any suggestions are welcomed. If possible please don't suggest not to use fork.

Best regards,
andrei

Quoted files follow:


http://rafb.net/p/eKup5083.html
------------------------------- use strict; use warnings; use File::Copy "copy"; while (1) { my $pid; $pid = fork(); if ($pid == 0) { copy("a", "b"); exit 0; } else { print "Pushing $pid\n"; print "\tWait : " . waitpid($pid, 0), "\n"; } }
-------------------------------

http://rafb.net/p/DkM9im72.html
-------------------------------
use strict; use warnings; use Win32; use File::Copy "copy"; while (1) { my $pid; $pid = fork(); if ($pid == 0) { copy("a", "b"); exit 0; } else { print "Pushing $pid\n"; print "\tWait : " . waitpid($pid, 0), "\n"; } }

-------------------------------

Replies are listed 'Best First'.
Re: Lots of memory usage with fork&copy
by Corion (Patriarch) on Apr 24, 2007 at 12:18 UTC

    Using fork on Windows Perl is almost every time an error because fork is merely emulated with threads and threads are not always working properly with Perl. Maybe you can tell us what problem you are trying to solve by forking instead of serially copying the files.

      I'm copying different files on different locations, some are over internet (on windows shares) and sometimes the copy process takes some time or the connections fails.
      Because of this lags I've got to do things in parallel. I have to copy after another process does its job so I have to have some kind of synchronisation.
      I could also execute another .pl everytime but I don't think that that is very efficient.

      I watched the number of threads and it doesn't grow more than 2, so it's ok. Probably it doesn't free the memory on exit&waitpid. Any ideas where exactly to report this as a bug? Directly to activestate's site or elsewhere?

      Thanks for your replies,
      andrei
        I'm copying different files on different locations, some are over internet (on windows shares)

        Red flag on the play! You're copying files over the internet using Windows shares? That's wrong in a so many ways - performance, security, sanity... Stop right now and pick a new tool - rsync, scp, HTTP and FTP all spring to mind. All have solid Windows implementations which should be fairly easy to get running concurrently.

        -sam

Re: Lots of memory usage with fork&copy
by tbone1 (Monsignor) on Apr 24, 2007 at 12:15 UTC
    Hm, I would guess that this has something to do with the Windows implementation of Perl. I just tried this on my old G4 laptop (running OS X 10.4, and BSD under the hood) and it works fine, just like Linux.

    --
    tbone1, YAPS (Yet Another Perl Schlub)
    And remember, if he succeeds, so what.
    - Chick McGee

Re: Lots of memory usage with fork&copy
by BrowserUk (Patriarch) on Apr 27, 2007 at 09:01 UTC

    I just ran both scripts on my system copying a 32 MB file locally. I don't have a handy VPN 'd windows share, but it's hard to see how that would make any difference to the perl code action/performance.

    In both cases, I saw zero memory growth. For the first script, memory fluctuated between 3.18MB and 3.27MB. For the second, between 3.680MB & 3.8MB. These flutations are simply snapshots as the process creates and destroys threads and are perfectly normal.

    Which version of perl are you using?

    Does it make any difference whether you copy the file locally?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Here is a image of resources usage:
      http://img266.imageshack.us/img266/6838/perlfh2.png
      The memory grows until I suspend the process (or windows says: Enough memory, Rest in Peace! ;) ).

      The "use Win32;" follows: http://img266.imageshack.us/my.php?image=perlokvu7.png

      As i said before "I run this scripts with perl v5.8.8 on Windows XP. (Binary build 817, ActiveState, 20 Mar 2006)."

      I've tried it with both an empty file and a 49MB file. With and empty file the memory usage grows faster because it forks faster.

      I've run the scripts on local files.

      Edit: g0n - linkified web links

        FYI: The use of a fileshare and File::Copy are both incidental to the problem. The following one-liner demonstrates the memory growth under both 5.8.8 and 5.9.5.

        The second one-liner also demonstrates that use Win32; definitely has the effect of slowing the growth rate:

        ## leaks rapidly. The sleep just avoids a fork bomb. \as817\perl\bin\perl.exe -le"while(Win32::Sleep 10){$p=fork and waitpi +d($p,0) or print(qq[kid:$$]),exit}" ## This leaks more slowly \as817\perl\bin\perl.exe -mWin32 -le"while(Win32::Sleep 10){$p=fork an +d waitpid($p,0) or print(qq[kid:$$]),exit}"

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        Sorry. That's the problem with coming to the party late, you tend to miss what has gone before. In this case, the reference to 5.8.8/AS817. I was using AS811/5.8.6 which I have stuck with in preference to 5.8.8 as I have encountered several anomolies with the latter.

        I just repeated my tests using AS817 and replicated your results exactly. I also tried it with 5.9.5 (current - 10 days), and found that it produces similar results to 5.8.8, so it looks like bug introduced since 5.8.6 is responsible. Worth raising a perl-bug report I think.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.