A first thing would be to make your test case(s) accessible to us. This means that you'll have to at least find out one combination of module versions that has the memory leak. Preferrably, these are all the latest module versions.

As a second step, I would try to eliminate as much of Catalyst and KiokuDB as possible, and call the offending part of the code in a tight loop as to exaggerate the memory consumption as far as possible.

From my cursory look at Catalyst::Model::KiokuDB::save_scope(), it calls another subroutine, ->setup_scope_guard(), which already has caveats about memory leaks. So I would continue from the code in there - most likely, some callback is closing over variables that it should release.

Are you sure that your diagnostic tools likel Devel::Monitor and Devel::Cycle are applicable resp. work? Personally, I prefer to override the DESTROY method of classes to track when an object gets destroyed:

my $old_destroy = \&Catalyst::Model::KiokuDB::DESTROY;
*Catalyst::Model::KiokuDB::DESTROY = sub {
    warn "$_[0] destroyed";
    goto &$old_destroy;
};
[download]

together with a block at the end of the main program telling me when global destruction starts.

I've edited my original post to make the code downloadable and written a step by step test case.

I've eliminate all but the defaults from Catalyst and KiokuDB by making the test case create a new application from scratch. The test case doesn't prune Catalyst further as it should (in theory) work just like that.

I've noticed the comment in Catalyst::Model::KiokuDB::setup_scope_guard() too. But Devel::Monitor claims that all variables declared in setup_scope_guard() and the callback in Scope::Guard are freed. Obviously, something isn't freed probably, but I'm starting to think it isn't an object, I'm looking for.

Devel::Monitor does something very similar to overriding DESTROY, only it does it by using some tie-magic, I think. But the result is the same; tracking when an object gets destroyed on-screen. Anyway, I'll try overriding DESTROY for a few candidates, just in case.

You can try Test::LeakTrace. If there's any XS involved you may also want Valgrind

How would Valgrind help with Perl data structures?

As far as I'm aware, Valgrind can only check whether a C program overwrites memory it has not allocated, and maybe can check whether a C program allocates memory and then loses the pointer(s) to that piece of memory. If you have a valid Perl scalar, Perl will keep a valid (C) pointer to it, and will release the memory during Global Destruction. If that's true, then Valgrind will/can not flag this Perl value as ever lost, even though it may be a memory leak in the sense that there is no Perl structure from the running set of Perl variables to access that piece of memory.

I recently spent quite a lot of time trying to find memory leak in Perl code, and finally ran it through Valgrind and discovered that it is YAML::XS. For sure it will not help with Perl structures, but I never said that.

Test::LeakTrace might have put me on the right track, and it looks like I can narrow it further down to KiokuDB:

use strict;
use warnings;
use Test::More;
use Test::Exception;
use Test::LeakTrace;
use KiokuX::Model;

my $model;
eval { $model = KiokuX::Model->new(dsn => 'couchdb::uri=http://127.0.0
+.1:5984/lb-local'); };
plan skip_all => 'Cannot connect to CouchDB' if ($@);

{
    no_leaks_ok {
        {
            my $scope = $model->new_scope;
        }
    } 'new_scope must not leak';
}

done_testing;
[download]

gives me:

ot ok 1 - new_scope must not leak (leaks 2 <= 0)
#   Failed test 'new_scope must not leak (leaks 2 <= 0)'
#   at KiokuX_Model.t line 17.
#     '2'
#         <=
#     '0'
# leaked SCALAR(0x4159810) from /usr/lib64/perl5/vendor_perl/5.12.4/Ki
+okuDB/LiveObjects.pm line 191.
#  190:
#  191:    $known->remove($scope);
#  192:
# SV = IV(0x4159808) at 0x4159810
#   REFCNT = 1
#   FLAGS = (IOK,pIOK)
#   IV = 0
# leaked ARRAY(0x4201180) from /usr/lib64/perl5/vendor_perl/5.12.4/Kio
+kuDB/LiveObjects.pm line 260.
#  259:
#  260:    $self->_known_scopes->insert($child);
#  261:
# SV = PVAV(0x4578968) at 0x4201180
#   REFCNT = 2
#   FLAGS = ()
#   ARRAY = 0x4580b40
#   FILL = 0
#   MAX = 3
#   ARYLEN = 0x0
#   FLAGS = (REAL)
#     Elt No. 0
#     SV = IV(0x4159808) at 0x4159810
#       REFCNT = 1
#       FLAGS = (IOK,pIOK)
#       IV = 0
1..1
# Looks like you failed 1 test of 1.
[download]

Valgrind also complained about something in Set::Object, so it might be the source of the problem. I'll keep digging.

I think I got my leak narrowed down now, and it looks like Set::Object::Weak is the problem. I've written the following short test script:

use strict;
use warnings;
use Config;
use Test::More;
use Test::LeakTrace;
use Set::Object;

{
    package Foo;
    use Moose;
    1;
}

{
    no strict;
    note join ' ', map {$Config{$_}} qw(osname archname);
    note 'perl version ', $];
    note $_,'-',${"${_}::VERSION"} for qw{Moose Set::Object Test::Leak
+Trace};
}

my $set;
{
    $set = Set::Object->new;
    no_leaks_ok {
        {
            my $obj = Foo->new;
            $set->insert($obj);
            $set->remove($obj);
        }
    } 'Testing Set::Object for leaking';
}

{
    $set = Set::Object::Weak->new;
    no_leaks_ok {
        {
            my $obj = Foo->new;
            $set->insert($obj);
            $set->remove($obj);
        }
    } 'Testing Set::Object::Weak for leaking';
}

done_testing;
[download]

which gives me:

# linux x86_64-linux
# perl version 5.012004
# Moose-2.0202
# Set::Object-1.28
# Test::LeakTrace-0.13
ok 1 - Testing Set::Object for leaking (leaks 0 <= 0)
not ok 2 - Testing Set::Object::Weak for leaking (leaks 2 <= 0)
#   Failed test 'Testing Set::Object::Weak for leaking (leaks 2 <= 0)'
#   at Set_Object.t line 39.
#     '2'
#         <=
#     '0'
# leaked SCALAR(0xdad3a0) from Set_Object.t line 37.
#   36:            $set->insert($obj);
#   37:            $set->remove($obj);
#   38:        }
# SV = IV(0xdad398) at 0xdad3a0
#   REFCNT = 1
#   FLAGS = (IOK,pIOK)
#   IV = 0
# leaked ARRAY(0x1163cd0) from Set_Object.t line 36.
#   35:            my $obj = Foo->new;
#   36:            $set->insert($obj);
#   37:            $set->remove($obj);
# SV = PVAV(0x1394b00) at 0x1163cd0
#   REFCNT = 2
#   FLAGS = ()
#   ARRAY = 0x13ac740
#   FILL = 0
#   MAX = 3
#   ARYLEN = 0x0
#   FLAGS = (REAL)
#     Elt No. 0
#     SV = IV(0xdad398) at 0xdad3a0
#       REFCNT = 1
#       FLAGS = (IOK,pIOK)
#       IV = 0
1..2
# Looks like you failed 1 test of 2.
[download]

Can anyone else here verify this? And if not, could I kindly ask you to post the top 5 lines of the output (the os, arch and versions)?