Re^3: Getting a memory dump

It is possible I could try running it in Win-XP but it would take quite a bit of setup.

Hm. It's possible that the code could be adapted for use on Linux, but I do not have that expertise.

The basic idea is that I have a module (Devel::MemWatch) that hooks the DB::DB() interface and pushes the current caller information (package/file/line no) onto a Thread::Queue as each line executes. It has a setable threshhold value above which it discards old information as it pushes new, effectively turning the queue into a circular buffer of the last N lines executed.

It also starts a background thread that wakes up every setable N millseconds and queries the process memory size from the system. If the memory expands beyond a setable limit, it dumps the circular buffer to stderr; doubles the memory limit, and goes back to monitoring.

The idea is that you get a dump of the last N lines that executed when the memory expanded beyond your preset limit. (And again every time it doubles again). Thus, you can quickly get an idea of what code was executing when the memory started to snowball.

The (unbelievably crude but functional) Win32 code looks like this:

#! perl -slw
use strict;
package Devel::MemWatch;
use threads;
use Thread::Queue;
use Win32::API::Prototype;

our %OPTS = (
    LINES => 100,
    THRESHHOLD => 100 *1024,
    FREQUENCY => 2000,
);

sub import {
    for ( @_[ 1 .. $#_ ] ) {
        my( $key, $value ) = split '=';
        $OPTS{ $key } = $value || '1';
    }
    warn "@{[ %OPTS ]}\n";
}

ApiLink( 'Kernel32', q[
    HANDLE GetCurrentProcess( void )
] ) or die $^E;

ApiLink( 'Kernel32', q[
    BOOL GetProcessHandleCount( DWORD h, LPDWORD c)]
) or die $^E;

ApiLink( 'PSAPI', q[
    BOOL GetProcessMemoryInfo(
        HANDLE Process, LPVOID p, DWORD cb
    )
] ) or die $^E;

sub getProcessMemoryInfo {
    my $hProcess = GetCurrentProcess( [] );
    my $buf = pack 'L10', 40, (0) x 9;
    my $size = 40;

    if( GetProcessMemoryInfo( $hProcess, $buf, $size ) ){
#        warn __LINE__;
        my( @MemStats ) = unpack( "L10", $buf );
        my $memusage = int( $MemStats[3] / 1024 );
        my $peak_memusage = int( $MemStats[2] / 1024 );
        my $vmsize = int( $MemStats[8] / 1024 );
        return wantarray ? ( $memusage, $peak_memusage, $vmsize ) : $m
+emusage;
    }
    else {
        die "GPMI: $^E";
    }
}

my $Q;

BEGIN{ $Q = new Thread::Queue }

sub DB::DB {
#    warn caller;
    my( $package, $file, $line ) = caller;
    scalar $Q->dequeue if $Q->pending > $OPTS{ LINES };
    $Q->enqueue( "$package $file $line" );
    return;
}

sub _dump {
    warn $Q->dequeue() . "\n" while $Q->pending;
}

async {
    warn "Watchthread started\n";
    while( Win32::Sleep( $OPTS{ FREQUENCY } ) ) {
        my $mem = getProcessMemoryInfo();
#        warn "Watchthread awoke:$mem\n";
        if( $mem > $OPTS{ THRESHHOLD } ) {
            warn "Memsize: $mem\n";
            $OPTS{ THRESHHOLD } *= 2;
            _dump()
        }
    }
}->detach;

1;
[download]

Usage is:

perl -d:MemWatch=LINES=100,THRESHHOLD=100*1024,FREQUENCY=2000 yourscri
+pt.pl
[download]

Where

LINES: no. of caller traces held in the circular buffer.
THRESHHOLD is in Kbytes. (default 100MB)
FREQUENCY (at which the watchthread awakes) is in milliseconds. (Default:2 seconds)

The output looks like:

C:\test>perl -d:MemWatch=FREQUENCY=5000 junk9.pl
FREQUENCY 5000 LINES 100 THRESHHOLD 102400
Watchthread started
Memsize: 103156
main junk9.pl 8
main junk9.pl 9
main junk9.pl 9
main junk9.pl 8
main junk9.pl 9
main junk9.pl 9
...
[download]

Maybe you can adapt it to your needs?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^3: Getting a memory dump Select or Download Code

Replies are listed 'Best First'.
Re^4: Getting a memory dump by awy (Acolyte) on Nov 23, 2007 at 15:54 UTC
Thanks, that looks like it could be useful. As far as I can see, the Win32-specific bit is just the guts of the getProcessMemoryInfo sub (and the sleep). That should be easy to adapt.	[reply]
Re^4: Getting a memory dump by awy (Acolyte) on Nov 26, 2007 at 08:18 UTC
This looked like a useful idea but I have been unable to get it work for me yet. Adopting the calls for getProcessMemoryInfo on Linux was easy enough. Something like: `open(PID, "/proc/$$/statm"); return (split(' ', <PID>))[5] * 4;` [download] but I found that the overhead of handling every statement with DB::DB was far too high. I tried to adjust the script to use DB::sub and just record subroutine calls. I need to include something in my DB::sub function that handles the recursion caused by the calls to `$Q->pending` etc. but so far I have failed to come up with a mechanism that both breaks the recursion and actually keeps on executing. Any suggestions would be appreciated. `my $handling = 0; sub DB::sub { return &$DB::sub if $handling; $handling = 1; scalar $Q->dequeue if $Q->pending > $OPTS{ LINES }; $Q->enqueue( $DB::sub ); $handling = 0; return &$DB::sub; }` [download]	[reply] [d/l] [select]
Re^5: Getting a memory dump by BrowserUk (Patriarch) on Nov 26, 2007 at 09:04 UTC
See this blog entry by Jonathan T. Rockway which shows his progress on the exact same problems you've encountered. He's gone much further than I have with this and his code snippets shoudl be useful to you. I have a few ideas myself also. I'll post again if I get anything useful. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]