Ok, to start with, what I observed was gradual growth in the VmSize of my C++ application when I repeatedly ran an embedded perl component.

I thought that the leak was large enough that I might be able to figure out what was leaking just by looking at a memory dump. (This was partly because I first discovered the leak in a production build, which was lacking debugging symbols and had been running for 10 hours, and I didn't want to kill it and wait for it to grow again.)

So I wrote a little tool in C to dump out the memory image. I thought that all I had to do was cat out /proc/pid/mem, but that doesn't work. Though come to think of it, I'm not sure what that should do, considering that the address space isn't contiguous. It turns out you can only read that file if you're ptracing the process, so that's what I did. I now have a handle little memcat utility that can take a pid, a starting address, and a length.

I apologize if this is veering too far from perl.

I did a diff of two copies of /proc/pid/maps to find a good starting point (a memory region that was expanding. There was only one, so it was easy to find.)

Next I ran strings on the output of memcat, piped through a histogram generator -- something like perl -lne '$c{$_}++;END{print "$c{$_} $_" foreach sort {$c{$a}<=>$c{$b}} keys %c }' (see, this is related to perl! Honest!)

This showed that I was leaking many, many copies of strings like $RX0002::time and ScPerl.cpp and \n]*), all of which could be traced to my embedded perl stuff. I discovered that I still didn't fully understand the whole refcounting scheme that perl uses. I'm still confused about how to manage line numbers properly -- I have one chunk of code where I want to temporarily change the file and line number to my C++ source file (because it calls a bunch of perl API points that can trigger warnings.) I basically do:

  char* oldFILE = CopFILE(PL_curcop);
  CopFILE_set(PL_curcop, __FILE__);
  ...
  CopFILE_free(PL_curcop);
  CopFILE_set(PL_curcop, oldFILE);
[download]

The question is, how do I safely dispose of oldFILE now? I am now calling Safefree(oldFILE), but I don't understand this stuff well enough to know if this is correct.

Back to the story. I managed to eliminate those extremely common strings. What remains is two somewhat less common chunks of memory. One looks like this:

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
+&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
[download]

The other, something like this:

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
+&&&&&&&&&&&&&&
[download]

The number of copies of these chunks climbs slowly throughout the lifetime of the app (much more slowly than the previous leaks.) So I wandered around telling everybody "our application is leaking ampersands. It looks like it will run out after three or four days and crash."

Anyone know of a better, hopefully less handwritten, tool chain for this sort of thing? I ran valgrind, but it doesn't seem to pick up memory leaks within perl even when I set PL_perl_destruct_level = 1; just before calling perl_destruct(my_perl); perl_free(my_perl);. Am I not doing that part right?