As noted in Perl coredump analysis tool ?, I needed a minimal impact tool to monitor a runaway application, and there didn't appear to be any solutions at hand. I'm pleased to announce I've just uploaded Devel::STrace, along with Devel::RingBuffer and IPC::Mmap, which together provided the solution to my problem, and hopefully help solve similar problems for others in future. (If those links aren't yet active, check the homepage)

Update:
CPAN informs me I've committed a faux pas: the IPC::Mmap namespace has been registered to someone else, but no module ever uploaded. <sigh/> So I'll likely need to revise these packages a bit in the next few days.

Update 2:
I'm saved! The owner of "IPC::Mmap" has agreed to turn it over to me. Many thanks for the assistance of the good keepers of CPAN/PAUSE for their help.

The solution begins with IPC::Mmap, which provides a minimal interface to memory mapped files as a shared memory solution on both Win32 (via Win32::MMF), and on POSIX platforms implementing mmap() (I've only tested LInux and Solaris at present).

On top of that foundation, Devel::RingBuffer implements a set of ringbuffers, plus some command/control structures. Devel::STrace supplies multithread/multiprocess friendly implementations of DB::DB() and DB::sub(), such that each thread of each process gets its own ringbuffer and command/control structure. Locking is minimized (I call it "cavalier concurrency control"), and much of the code is XS/C to minimize performance impact (though the impact is still significant, comparable to running with "-d" in NonStop mode).

A Devel::STrace::Monitor package is also provided to permit external monitoring applications to access the ringbuffers, and , finally, a small monitor script, plstrace.pl, implements the monitor application which attaches to the monitored application and dumps out the current ringbuffer contents at specified intervals.

E.g., to monitor an application: 

perl -d:STrace someapp.pl

[download]
which will dump the name of the ringbuffer file/namespace to STDERR upon creation; that name is then plugged into 
plstrace.pl -s <sample> -d <duration> -f <file>

[download]
which will print the current ring buffer contents every <sample> seconds for <duration>. (There are numerous tuning parameters set via environment variables; refer to the package docs for details)

As a first release, there remain some holes:

Using an early version of this setup, I was able to diagnose and fix the original pesky runaway problem within 30 minutes of starting the application under "-d:STrace", after nearly 2 weeks of frustration trying to isolate the problem. The plstrace output pinpointed exactly where things were going openloop, without having to wade through mountains of debugger output, and the associated huge performance hit.

Perhaps as importantly, along the way I realized that this substrate provided intriguing possibilities for developing decoupled, multithreaded/multiprocess-friendly debuggers...

I had recently posted to p5p to verify it was possible to make $DB::single threads::shared (for purposes of developing a more user-friendly debugger for multithreaded/multiprocess apps). While the answer was yes, the measured impact of share()ing $DB::single was extremely large.

As I started developing the ringbuffer structures for Devel::STrace, I realized it would be a simple matter of adding some command/control hooks that can be used to build debuggers which run nearly entirely separately from the debugged application. With such hooks, it would be possible to

If you've ever peered into perl5db.pl, or ptkdb, you'll realize how "intimate" those debuggers are with the debugged app. Devel::RingBuffer plus an appropriate derivative of Devel::STrace break that tight coupling, thus permitting, e.g., a Perl/Tk debugger GUI app to attach to a running Gtk perl app, debug as needed, and then detach to let the application run. (Hopefully, this solution could apply to CGI or mod_perl apps as well ?). And, unlike Devel::ebug, which relies on TCP sockets communications between debugger and debugged app, the command/control interface is as simple - and fast - as reading a few bytes of memory.

Update:
It appears an attachable debugger may have some supporters; wish I had the time...

Alas, my spare cycles for the effort of implementing these notions are currently at a premium. Any volunteers ?

Back to Meditations