I guess what you are saying is this is basically just how memory management is in PERL?

It's not: "how memory management is in Perl"; it's: how memory management is!

In general, it is expensive for a process to request more memory from the OS; so once a process (via your C compiler memory management functions) have requested memory from the OS, they are reluctant to give it back because it would be expensive to re-request it from the OS again. Better to simply keep it around until the next time it is needed.

This is, for the most part, the way all memory managers work.

How else would you deal with a long running application using threads not eating up all the memory?

I'll say it again. There is no problem with running many dozens of concurrent threads. With 512MB, based on the numbers you gave above, you should be able to run ~50 threads concurrently without problems.

And so long as no more than 50 are ever running at any given time; you should be able to start & end thousands of threads without ever breaking that same limit.

But whether it is a good idea to do so is a different question that can only be answered by seeing your code.

It is not the number of threads you start; nor how long the process runs for that is the limiting factor; but rather the number of them that are running at the same time. It's not rocket science to see that the more you run concurrently; the more memory it will use. Use too many at the same time and you'll run out of a finite resource.

Threads are killed using a signal which calls threads->exit().

Killing threads is a really bad idea; and IS likely to cause memory leaks.

I was looking through lists of "memorable nodes", collected on some monks' home pages, that's why I came here to such old discussion.

I can't quite agree:

It's not: "how memory management is in Perl"; it's: how memory management is!

In general, it is expensive for a process to request more memory from the OS; so once a process (via your C compiler memory management functions) have requested memory from the OS, they are reluctant to give it back

It looks like, memory requested by XS code is eventually returned to OS (right?). And memory, allocated in separate threads, upon their completion, is returned, too. (That's why if I want long-running program to have sane memory-footprint, I'd place memory hungry code in a thread. Or even, cf. my last question about child processes and IPC, to separate it completely.)

But, "pure Perl" mono-thread is, indeed, reluctant to give anything back.

Here's on Win32 5.020 (your numbers may vary, and sorry for a bit barbaric "memory monitoring code". CPAN Windows-compatible modules seem to fail with threads):

use strict;
use warnings;
use threads;
use PDL;

PDL::no_clone_skip_warning;

sub mem {
    qx{ typeperf "\\Process(perl)\\Working Set" -sc 1 } 
        =~ /(\d+)\.\d+\"$/m;
    ( my $s = $1 ) =~ s/(\d{1,3}?)(?=(\d{3})+$)/$1,/g;
    printf "%-30s: %12s\n", @_, $s
}

mem 'initially';
my $p = zeroes 50_000_000;
mem 'we\'ve made a huge piddle!';
undef $p;
mem 'and now it\'s gone';

sub test_arr {
    my @a = 1 .. ${ \10_000_000 };
    mem 'we\'ve made a huge array!';
    @a = undef;
    mem 'and now it\'s gone';
}

async( \&test_arr )-> join;
mem 'and now a thread is gone, too';
print "\nbut let\'s try it in main thread!\n\n";
test_arr;
mem 'finally';
[download]

The output:

initially                     :   16,846,848
we've made a huge piddle!     :  418,107,392
and now it's gone             :   17,321,984
we've made a huge array!      :  628,912,128
and now it's gone             :  628,928,512
and now a thread is gone, too :   20,258,816

but let's try it in main thread!

we've made a huge array!      :  625,430,528
and now it's gone             :  625,430,528
finally                       :  625,430,528
[download]

It looks like, memory requested by XS code is eventually returned to OS (right?).

No. Not in general.

Under Windows(*), if you request a large (from memory it was >1MB; but that may have changed) contiguous chunk of memory; then instead of calling the C runtime malloc(), it goes direct to the OS and uses VirtualAlloc() to allocate a distinct chunk of virtual memory from the system memory pool rather than taking a chunk from the process' heap.

Because that single large allocation is only used for that single purpose -- a very large string or the base allocation of a large array or hash -- when the owning Perl entity (scalar/hash/array) is freed, the entire virtual allocation can be returned to the OS.

This includes the stacks allocated to threads which under the defaults, often predominates the memory used by a thread -- see threads module stack_size parameter for how to vastly reduce the memory consumption of threads. Most of my threaded coded uses nothing more than a 4kB stack rather than the default of 1MB.

But all allocations less than that 1MB limit -- which for most Perl programs is the vast majority of allocations -- come from process heap via the CRT malloc(), and they are never returned to the OS until the process ends.

Your program is unusual in that you are apparently using both threads -- which tend to allocate large chunks for stacks -- and pdl with very large piddles, which are allocated as single contiguous chunks. That combination when running on windows gives you a distorted view of Perl's ability to return memory to the OS.

*I seem to recall seeing mention that Perl now(v5.1something) does a similar thing -- bypassing malloc() for large contiguous allocations -- under *nix; but I couldn't swear to it.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.

In the absence of evidence, opinion is indistinguishable from prejudice.