in reply to OS memory reclamation with threads on linux

Can we trust top/ps to really report what is going on.

Not usually; Linux allows memory overcommit. As I understand it (and simplifying somewhat), any application can request as much memory as is present on the machine (the VIRT column in top), but only the RES column is actually in use at any time. If all of the RES memory exceeds physical memory, it's swap time.

Without knowing what you watched grow and shrink in top, it's difficult to explain what happened.

Replies are listed 'Best First'.
Re^2: OS memory reclamation with threads on linux
by zentara (Cardinal) on Aug 27, 2008 at 16:53 UTC
    I watch the %MEM columns in top/ps. It's been confusing testing, because when there is a small percent difference between the peak and normal mem usage, sometimes the kernel shows %MEM drop, other times not. If the diff is big, the kernel immediately returns it.

    My real question is what are people talking about when they claim their threaded apps are increasing in memory. Do I just say "forget it.... the kernel will grab it if needed", or "you have a thread design problem"

    I'm also wondering if this has different behavior with the older 5.8 threads vs. the latest threads (which seem to run better), and/or kernel versions.

    It seemed like just last week that this was a constant problem with Perl threads, and now suddenly it seems like a great thing, where you can reclaim memory from mem-intensive Perl scripts, by putting the mem-intensive stuff into a thread.

    I'm not really a human, but I play one on earth Remember How Lucky You Are
[reply]
      I watch the %MEM columns in top/ps

      Well, that's proportional to RES, it's the amount of physical memory used by the app at that moment (including all loaded libs and possibly shared memory). It only reflects your apps "memory usage" if the box isn't under memory pressure (i.e. too much running at once). If your box *is* under memory pressure, then RES and %MEM will relate to what the OS is running at that moment - that figure will go down when bits of your app is paged out for something else to run, even though your app is still "using the same amount of memory" (i.e. it hasn't free'd anything).

      My real question is what are people talking about when they claim their threaded apps are increasing in memory.

      My first guess would be that they've just got a good old-fashioned memory leak.

      The OS generally makes good decisions regarding memory use. The best advice is to not worry about it too much until you have a problem app, unless you're into learning things (which is cool).

[reply]

In Section Meditations