in reply to Cleaning up threads so they do not leak memory. (Win32)

Stack (C time) space allocated for use by the threads can be returned to the OS, as a separate virtual allocation is made for each thread when it is created, and return to the system when the thread is destroyed.

Heap space used by the threads comes from the same, single, virtual allocation made by the process at startup, so any expansion of the heap for use by the threads cannot be returned to the OS. However, it is available for re-use by the main thread or any subsequent threads that may be created. Technically, this is not a leak.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Cleaning up threads so they do not leak memory. (Win32)
by slloyd (Hermit) on Feb 23, 2006 at 13:41 UTC
    My problem with that is that if I write a threaded program that stays running (like a proxy server or a web server) the longer it runs the more memory it takes. Pretty soon, it is using all the memory on the system. Is there a way to force the memory to be returned to the OS?

    -------------------------------
    by me
    http://www.basgetti.com
    http://www.kidlins.com

[reply]
      My problem with that is that if I write a threaded program that stays running (like a proxy server or a web server) the longer it runs the more memory it takes.

      Unfortunately, there is still some memory used by the, (unnecessary in most cases!), cloning process that is performed when a thread is spawned, that Perl's memory management never recovers. This is the output from a modified version of your OP code that loops 50 times over the thread spawning/death measuring the memory at each stage: 

      C:\Perl\src\perl-5.8.8\win32>..\perl -mlib=\perl\site\lib \test\532177
+.pl
StartMem: 3344 MB
Pass 1  MidMem: 18340 MB        EndMem: 5228 MB
Pass 2  MidMem: 18380 MB        EndMem: 5976 MB
Pass 3  MidMem: 18384 MB        EndMem: 6900 MB
Pass 4  MidMem: 18396 MB        EndMem: 6896 MB
Pass 5  MidMem: 18396 MB        EndMem: 7356 MB
Pass 6  MidMem: 18392 MB        EndMem: 7796 MB
Pass 7  MidMem: 18392 MB        EndMem: 7984 MB
Pass 8  MidMem: 18392 MB        EndMem: 7928 MB
Pass 9  MidMem: 18392 MB        EndMem: 8064 MB
Pass 10 MidMem: 18400 MB        EndMem: 7968 MB
Pass 11 MidMem: 18400 MB        EndMem: 8928 MB
Pass 12 MidMem: 18400 MB        EndMem: 8544 MB
Pass 13 MidMem: 18400 MB        EndMem: 8612 MB
Pass 14 MidMem: 18400 MB        EndMem: 8736 MB
Pass 15 MidMem: 18400 MB        EndMem: 8620 MB
Pass 16 MidMem: 18400 MB        EndMem: 8540 MB
Pass 17 MidMem: 18400 MB        EndMem: 8256 MB
Pass 18 MidMem: 18404 MB        EndMem: 8968 MB
Pass 19 MidMem: 18404 MB        EndMem: 8652 MB
Pass 20 MidMem: 18404 MB        EndMem: 8580 MB
Pass 21 MidMem: 18396 MB        EndMem: 9028 MB
Pass 22 MidMem: 18408 MB        EndMem: 8852 MB
Pass 23 MidMem: 18404 MB        EndMem: 9272 MB
Pass 24 MidMem: 18404 MB        EndMem: 8812 MB
Pass 25 MidMem: 18404 MB        EndMem: 9012 MB
Pass 26 MidMem: 18404 MB        EndMem: 8980 MB
Pass 27 MidMem: 18412 MB        EndMem: 9480 MB
Pass 28 MidMem: 18408 MB        EndMem: 9524 MB
Pass 29 MidMem: 18400 MB        EndMem: 9352 MB
Pass 30 MidMem: 18408 MB        EndMem: 9508 MB
Pass 31 MidMem: 18404 MB        EndMem: 9240 MB
Pass 32 MidMem: 18404 MB        EndMem: 9084 MB
Pass 33 MidMem: 18404 MB        EndMem: 9324 MB
Pass 34 MidMem: 18412 MB        EndMem: 9368 MB
Pass 35 MidMem: 18416 MB        EndMem: 9240 MB
Pass 36 MidMem: 18420 MB        EndMem: 9500 MB
Pass 37 MidMem: 18412 MB        EndMem: 9532 MB
Pass 38 MidMem: 18412 MB        EndMem: 9680 MB
Pass 39 MidMem: 18424 MB        EndMem: 9596 MB
Pass 40 MidMem: 18424 MB        EndMem: 10116 MB
Pass 41 MidMem: 18424 MB        EndMem: 10256 MB
Pass 42 MidMem: 18416 MB        EndMem: 10092 MB
Pass 43 MidMem: 18428 MB        EndMem: 10028 MB
Pass 44 MidMem: 18428 MB        EndMem: 10060 MB
Pass 45 MidMem: 18428 MB        EndMem: 10388 MB
Pass 46 MidMem: 18420 MB        EndMem: 10248 MB
Pass 47 MidMem: 18428 MB        EndMem: 10348 MB
Pass 48 MidMem: 18420 MB        EndMem: 10320 MB
Pass 49 MidMem: 18428 MB        EndMem: 10084 MB
Pass 50 MidMem: 18420 MB        EndMem: 10540 MB

      [download]

      As you can see, whilst the vast majority of the memory consumed by the spawned threads is recovered after each pass, there is a residual growth. And even though the residual sometimes falls back further on one pass than it did the previous pass, there is an inexorable upwards trend that cannot be explained by heap growth or memory fragmentation. I think that does consitute a leak.

      This is one reason why I continue to advocate using a few long-running threads rather than spawning threads that "do & die". I prefer long running threads that wait on a queue for work, do the work, then go back to get another batch to do anyway, but given the continued leaks during the cloning process, it seems to be the only way to get long running threaded Perl processes to work properly.

      Depending upon what you are doing and how your code is currently structured, it can be quite easy to convert "do & die" code to the "wait/run/wait" model, and it generally speaking tends to be more efficient anyway. I'd be willing to try and help you make the transition.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[reply]
[reply]

In Section Seekers of Perl Wisdom