It looks like, memory requested by XS code is eventually returned to OS (right?).

No. Not in general.

Under Windows(*), if you request a large (from memory it was >1MB; but that may have changed) contiguous chunk of memory; then instead of calling the C runtime malloc(), it goes direct to the OS and uses VirtualAlloc() to allocate a distinct chunk of virtual memory from the system memory pool rather than taking a chunk from the process' heap.

Because that single large allocation is only used for that single purpose -- a very large string or the base allocation of a large array or hash -- when the owning Perl entity (scalar/hash/array) is freed, the entire virtual allocation can be returned to the OS.

This includes the stacks allocated to threads which under the defaults, often predominates the memory used by a thread -- see threads module stack_size parameter for how to vastly reduce the memory consumption of threads. Most of my threaded coded uses nothing more than a 4kB stack rather than the default of 1MB.

But all allocations less than that 1MB limit -- which for most Perl programs is the vast majority of allocations -- come from process heap via the CRT malloc(), and they are never returned to the OS until the process ends.

Your program is unusual in that you are apparently using both threads -- which tend to allocate large chunks for stacks -- and pdl with very large piddles, which are allocated as single contiguous chunks. That combination when running on windows gives you a distorted view of Perl's ability to return memory to the OS.

*I seem to recall seeing mention that Perl now(v5.1something) does a similar thing -- bypassing malloc() for large contiguous allocations -- under *nix; but I couldn't swear to it.